Tactile signal encoding and decoding processing methods and related devices

Abstract

The application discloses a kind of haptic signal encoding processing method, decoding processing method and related equipment, belong to haptic codec field, the haptic signal encoding processing method of the embodiment of the application includes: obtaining at least one of the key frame sequence and first signal corresponding to each haptic signal in multiple haptic signals, the key frame sequence is obtained based on second signal, first signal is obtained based on target residual signal and third signal, target residual signal is obtained based on second signal and key frame sequence, second signal and third signal are obtained based on to haptic signal, and the frequency of second signal is less than the frequency of third signal;At least one of the following is performed: first masking processing is carried out to key frame sequence based on first masking model, and the signal after first masking processing is encoded to obtain first encoding sequence;Second masking processing is carried out to first signal based on second masking model, and the signal after second masking processing is encoded to obtain second encoding sequence.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of haptic coding, and particularly relates to a haptic signal encoding processing method, a haptic signal decoding processing method and related equipment. BACKGROUND

[0002] In the encoding process of a haptic signal (which includes but is not limited to a vibration texture perception signal), a plurality of single-point haptic signals are usually encoded one by one to realize the encoding of a multi-point haptic signal. However, due to the masking effect between adjacent haptic signals, the haptic signal after the one-by-one encoding of the single-point haptic signal has a certain redundancy. Therefore, the prior art has the problem of low compression rate of the haptic signal. SUMMARY

[0003] The embodiments of the application provide a haptic signal encoding processing method, a haptic signal decoding processing method and related equipment, which can solve the problem of low compression rate of the haptic signal.

[0004] In a first aspect, a haptic signal encoding processing method is provided, applied to an encoding end, and including:

[0005] obtaining at least one of a key frame sequence and a first signal corresponding to each of a plurality of haptic signals, wherein the key frame sequence is obtained based on a second signal, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on the second signal and the key frame sequence, the second signal and the third signal are obtained based on low-pass filtering processing on the haptic signal, and the frequency of the second signal is less than the frequency of the third signal;

[0006] performing a target operation; wherein the target operation includes at least one of:

[0007] performing first masking processing on the key frame sequence based on a first masking model, and encoding the signal after the first masking processing to obtain a first encoding sequence;

[0008] performing second masking processing on the first signal based on a second masking model, and encoding the signal after the second masking processing to obtain a second encoding sequence.

[0009] In a second aspect, a haptic signal decoding processing method is provided, applied to a decoding end, and including:

[0010] decoding the second encoding sequence to obtain a plurality of fourth signals;

[0011] performing block processing on each of the fourth signals to obtain at least one signal block, each signal block corresponding to a time interval;

[0012] determining a primary signal block and a secondary signal block corresponding to each time interval;

[0013] for each of the time intervals, performing a de-occlusion on the secondary signal block based on the primary signal block to obtain a fifth signal, the fifth signal including the primary signal block and the de-occluded secondary signal block;

[0014] based on the fifth signal, obtaining a multi-channel decoded signal;

[0015] wherein the second encoded sequence is obtained based on a first signal corresponding to each of the multi-channel haptic signals, the first signal being obtained based on a target residual signal and a third signal, the target residual signal being obtained based on a second signal, the second signal and the third signal being obtained based on low-pass filtering the haptic signals, and the second signal having a lower frequency than the third signal.

[0016] In a third aspect, a haptic signal encoding processing apparatus is provided, applied to an encoding end, and including:

[0017] an obtaining module, configured to obtain at least one of a key frame sequence and a first signal corresponding to each of the multi-channel haptic signals, the key frame sequence being obtained based on a second signal, the first signal being obtained based on a target residual signal and a third signal, the target residual signal being obtained based on the second signal and the key frame sequence, the second signal and the third signal being obtained based on low-pass filtering the haptic signals, and the second signal having a lower frequency than the third signal;

[0018] an executing module, configured to execute a target operation, wherein the target operation includes at least one of:

[0019] performing first occlusion processing on the key frame sequence based on a first occlusion model, and encoding the signal after the first occlusion processing to obtain a first encoded sequence;

[0020] performing second occlusion processing on the first signal based on a second occlusion model, and encoding the signal after the second occlusion processing to obtain a second encoded sequence.

[0021] In a fourth aspect, a haptic signal decoding processing apparatus is provided, applied to a decoding end, and including:

[0022] a decoding module, configured to decode a second encoded sequence to obtain a plurality of fourth signals;

[0023] a block processing, configured to perform block processing on each of the fourth signals to obtain at least one signal block, each signal block corresponding to a time interval;

[0024] The first determining module is configured to determine a main signal block and a secondary signal block corresponding to each time interval;

[0025] The restoring module is configured to, for each time interval, perform a masking restoration on the secondary signal block based on the main signal block to obtain a fifth signal, the fifth signal including the main signal block and the masking-restored secondary signal block;

[0026] The processing module is configured to obtain a multi-path decoding signal based on the fifth signal.

[0027] The second encoding sequence is obtained based on each of the multi-path haptic signals and a first signal, the first signal being obtained based on a target residual signal and a third signal, the target residual signal being obtained based on a second signal, the second signal and the third signal being obtained based on low-pass filtering processing on the haptic signal, and the frequency of the second signal being less than the frequency of the third signal.

[0028] In a fifth aspect, an electronic device is provided, which includes a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the method according to the first aspect, or the programs or instructions being executed by the processor to implement the steps of the method according to the second aspect.

[0029] In a sixth aspect, an electronic device is provided, which includes a processor and a communication interface, wherein,

[0030] When the electronic device is an encoding end, the processor is configured to obtain at least one of a key frame sequence corresponding to each of the multi-path haptic signals and a first signal, the key frame sequence being obtained based on a second signal, the first signal being obtained based on a target residual signal and a third signal, the target residual signal being obtained based on the second signal and the key frame sequence, the second signal and the third signal being obtained based on low-pass filtering processing on the haptic signal, and the frequency of the second signal being less than the frequency of the third signal; and perform a target operation; the target operation including at least one of: performing a first masking processing on the key frame sequence based on a first masking model, and encoding the signal after the first masking processing to obtain a first encoding sequence; performing a second masking processing on the first signal based on a second masking model, and encoding the signal after the second masking processing to obtain a second encoding sequence.

[0031] When the electronic device is a decoding end, the processor is configured to decode the second encoding sequence to obtain a plurality of fourth signals; perform block processing on each of the fourth signals to obtain at least one signal block, each signal block corresponding to a time interval; determine a main signal block and a secondary signal block corresponding to each time interval; for each time interval, perform concealment restoration on the secondary signal block based on the main signal block to obtain a fifth signal, the fifth signal including the main signal block and the restored secondary signal block; and obtain a plurality of decoding signals based on the fifth signal; wherein the second encoding sequence is obtained based on a first signal corresponding to each of the plurality of haptic signals, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on a second signal, and the second signal and the third signal are obtained based on low-pass filtering processing on the haptic signal, and the frequency of the second signal is less than the frequency of the third signal.

[0032] In a seventh aspect, a haptic signal encoding and decoding system is provided, including an encoding end device and a decoding end device, the encoding end device being configured to perform the steps of the haptic signal encoding method of the first aspect, and the decoding end device being configured to perform the steps of the haptic signal decoding method of the second aspect.

[0033] In an eighth aspect, a readable storage medium is provided, the readable storage medium storing a program or instructions, the program or instructions being executed by a processor to implement the steps of the method of the first aspect or the steps of the method of the second aspect.

[0034] In a ninth aspect, a chip is provided, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being configured to run a program or instructions to implement the steps of the method of the first aspect or the steps of the method of the second aspect.

[0035] In a tenth aspect, a computer program / program product is provided, the computer program / program product being stored in a storage medium, the computer program / program product being executed by at least one processor to implement the steps of the method of the first aspect or the steps of the method of the second aspect.

[0036] In the embodiment of the present application, at least one of the key frame sequence corresponding to each of the plurality of haptic signals and the first signal is obtained, wherein the key frame sequence is obtained based on a second signal, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on the second signal and the key frame sequence, the second signal and the third signal are obtained based on low-pass filtering processing of the haptic signal, and the frequency of the second signal is less than the frequency of the third signal; a target operation is performed; wherein the target operation includes at least one of the following: performing first masking processing on the key frame sequence based on a first masking model, and encoding the signal after the first masking processing to obtain a first encoding sequence; performing second masking processing on the first signal based on a second masking model, and encoding the signal after the second masking processing to obtain a second encoding sequence. In this way, since at least one of the key frame sequence and the first signal is subjected to masking processing, the signal redundancy can be removed, and the efficiency of haptic signal compression is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of a haptic signal encoding processing method provided by the embodiment of the present application;

[0038] Figure 2 is an encoding framework diagram of a haptic signal encoding processing method provided by the embodiment of the present application;

[0039] Figure 3 is a flowchart of a haptic signal decoding processing method provided by the embodiment of the present application;

[0040] Figure 4 is a decoding framework diagram of a haptic signal decoding processing method provided by the embodiment of the present application;

[0041] Figure 5 is a structural diagram of a haptic signal encoding processing device provided by the embodiment of the present application;

[0042] Figure 6 is a structural diagram of a haptic signal decoding processing device provided by the embodiment of the present application;

[0043] Figure 7 is a structural diagram of a communication device provided by the embodiment of the present application;

[0044] Figure 8 is a structural diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0045] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly described below. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art are within the scope of the present application.

[0046] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" are generally a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three schemes, namely, scheme one: including A and not including B; scheme two: including B and not including A; scheme three: including A and including B. The character " / " generally represents that the objects before and after are in an "or" relationship.

[0047] The term "indication" in the present application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). The direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed or requested results, etc. in the sent indication. The indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or judging and determining the operations to be performed or the requested results according to the judgment result.

[0048] The encoding and decoding end corresponding to the haptic signal encoding and decoding method in the embodiments of the present application can be a terminal, which can also be called a terminal device or a user terminal (User Equipment, UE). The terminal can be a mobile phone, a tablet personal computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a personal digital assistant (Personal Digital Assistant, PDA), a palm computer, a netbook, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a mobile Internet device (Mobile Internet Device, MID), a wearable device (Wearable Device) or a vehicle-mounted device (VUE), a pedestrian terminal (PUE), etc. Wearable devices include smart watches, bracelets, earphones, glasses, etc. It should be noted that the specific type of terminal is not limited in the embodiments of the present application.

[0049] For the convenience of understanding, some contents related to the embodiments of the present application are described below:

[0050] Touch, as a unique sensory channel of human body, can effectively obtain complex information that is difficult to be perceived by other perception forms such as vision or hearing. Touch information refers to the perception of human body on shape, position, surface texture, object hardness, roughness and temperature, and mainly includes two categories: touch information based on muscle movement perception (hereinafter referred to as dynamic touch information) and touch information based on skin texture perception (hereinafter referred to as texture touch information). The object surface texture, friction, temperature and the like perceived by the mechanical stimulation receptors on the skin all belong to texture touch information. After being collected by a hardware system and texture touch coding, the texture touch signal is reconstructed by a specific touch reconstruction device to deliver the combination of vibration or stimulation pulses perceived by the receptors to the user. For example, a new type of touch feedback display screen can experience texture touch through finger sliding on the touch screen; a vibrating wave touch feedback disc can increase the vibration touch perception on the existing virtual reality game audio-visual senses; Google adds a touch game handle to the game platform, and the like. The development of these touch reconstruction systems cannot be separated from texture touch coding technology, and texture touch coding technology is an important basis for the development of these devices.

[0051] Texture touch coding methods mainly include three categories: statistical-based coding method, coding method based on perception model (Weber's law) and hybrid perception coding method. Among them, the statistical-based texture touch coding utilizes quantization and prediction model to reduce data redundancy. The coding method based on perception model realizes data compression by setting a difference threshold (discrimination sensitivity). The hybrid perception coding extracts the necessary attributes of the input signal by using the absolute threshold of perception and the masking phenomenon to realize perception compression. The perceptual vibration touch signal compression algorithm (PVC-SLP) based on sparse linear prediction adopts an accelerated sensitivity function (ASF) to construct a touch sensitivity model, which is currently selected as part of the IEEE P1918.1.1 touch coding standard.

[0052] The touch signal coding method provided by the embodiments of the present application will be described in detail in combination with the drawings and some embodiments and application scenarios.

[0053] In step 101, at least one of a key frame sequence corresponding to each of a plurality of touch signals and a first signal is obtained, wherein the key frame sequence is obtained based on a second signal, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on the second signal and the key frame sequence, the second signal and the third signal are obtained based on low-pass filtering processing on the touch signal, and the frequency of the second signal is less than the frequency of the third signal;

[0054] In the embodiments of the present application, each haptic signal has a corresponding key frame sequence and a first signal. The second signal can be understood as a low-frequency signal component of the haptic signal, and the third signal can be understood as a high-frequency signal component of the haptic signal.

[0055] Optionally, in some embodiments, the multi-channel spatially adjacent haptic signals and haptic spatial information can be acquired first, wherein the spatial information includes but is not limited to the following information: body region information, position information in the body region, and relative position information of each signal source; then the following process can be used to obtain the key frame sequence and the first signal corresponding to each haptic signal:

[0056] Low-pass filtering is performed on the multi-channel haptic signals to separate the low-frequency signal components and the high-frequency signal components that are easy for the human body to perceive, thereby obtaining the second signal and the third signal;

[0057] Each signal extreme value of each second signal is taken as a key frame to obtain the key frame sequence corresponding to each second haptic signal;

[0058] A reconstructed low-frequency signal is obtained through interpolation reconstruction, and a target residual signal is obtained by calculating the residual between the reconstructed low-frequency signal and the corresponding second signal;

[0059] The target residual signal is added to the corresponding third signal, so that the first signal, i.e., the high-frequency signal with a low-frequency residual, can be obtained.

[0060] Step 102, performing a target operation; wherein the target operation includes at least one of the following:

[0061] Performing first masking processing on the key frame sequence based on a first masking model, and encoding the signal after the first masking processing to obtain a first encoding sequence;

[0062] Performing second masking processing on the first signal based on a second masking model, and encoding the signal after the second masking processing to obtain a second encoding sequence.

[0063] In the embodiments of the present application, the masking mode can be selected as needed: for example, only the key frame sequence can be masked, only the first signal can be masked, or both the key frame sequence and the first signal can be masked. It should be understood that when the key frame sequence and the first signal are not masked, it can be understood that the normal encoding processing is not performed on the object to be masked.

[0064] Optionally, the first masking model includes, but is not limited to, a spatio-temporal joint masking effect model suitable for low-frequency signals. The second masking model includes, but is not limited to, a spatio-temporal joint masking effect model suitable for high-frequency signals. The spatio-temporal joint masking effect model represents the masking effect of the primary signal on the surrounding secondary signal under the action of the primary signal at different frequencies, expressed by a masking threshold coefficient. The amplitude of the primary signal is multiplied by the masking threshold coefficient and subjected to a corresponding assignment operation to obtain a masking threshold matrix. When the amplitude of the secondary signal adjacent to the primary signal in space is lower than the masking threshold, it indicates that the primary signal has a masking effect on the secondary signal, and the human being cannot or hardly perceives the secondary signal under the influence of the primary signal.

[0065] In the embodiments of the present application, at least one of the key frame sequence corresponding to each of the plurality of haptic signals and the first signal is obtained, wherein the key frame sequence is obtained based on a second signal, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on the second signal and the key frame sequence, the second signal and the third signal are obtained based on low-pass filtering processing of the haptic signal, and the frequency of the second signal is less than the frequency of the third signal; a target operation is performed; wherein the target operation includes at least one of the following: performing first masking processing on the key frame sequence based on a first masking model, and encoding the signal after the first masking processing to obtain a first encoding sequence; performing second masking processing on the first signal based on a second masking model, and encoding the signal after the second masking processing to obtain a second encoding sequence. In this way, since at least one of the key frame sequence and the first signal is subjected to masking processing, signal redundancy can be removed, and the efficiency of haptic signal compression is improved.

[0066] Optionally, in some embodiments, the first encoding sequence is obtained by performing first masking processing on the key frame sequence based on the first masking model, which includes:

[0067] determining a primary key frame sequence and a secondary key frame sequence in the key frame sequence corresponding to the plurality of haptic signals;

[0068] determining the first masking model according to the spatial position information corresponding to the primary key frame sequence and the spatial position information corresponding to the secondary key frame sequence;

[0069] performing first masking processing on the secondary key frame sequence according to the first masking model to obtain a target key frame sequence, the target key frame sequence including the primary key frame sequence and the masked secondary key frame sequence;

[0070] encoding the target key frame sequence to obtain a first encoding sequence.

[0071] In the embodiments of the present application, the first encoding sequence can be referred to as a low-frequency encoding sequence. The main key frame sequence can be understood as a key frame sequence corresponding to a haptic signal of a haptic sensitive point, or can be understood as a key frame sequence corresponding to a haptic signal with a large signal strength. The secondary key frame sequence is a key frame sequence corresponding to a plurality of haptic signals other than the main key frame sequence. In other words, in the embodiments of the present application, the main key frame sequence satisfies at least one of the following conditions:

[0072] The main key frame sequence is a key frame sequence corresponding to a haptic signal of a haptic sensitive point.

[0073] The main key frame sequence is a key frame sequence corresponding to a haptic signal with a signal strength greater than or equal to a preset intensity threshold.

[0074] Optionally, the method for determining the main key frame sequence can include the following methods:

[0075] 1. When the signal strengths are approximately the same, the main key frame sequence is determined according to whether the position of the haptic signal is a haptic sensitive point. For example, the key frame sequence corresponding to the haptic signal of the haptic sensitive point is taken as the main key frame sequence. The haptic sensitive point can be obtained by subjective testing, or can be defined according to specific devices and specific parts.

[0076] 2. When there is a large interpolation of signal strength, the main and secondary key frame sequences can be defined according to the strength of the signal. For example, a specified time interval can be determined first, and the key frames contained in the time interval are interpolated to obtain the recovered signal,

[0077] The recovered signals are integrated according to factors such as amplitude and power, and the integral results are sorted in descending order. According to the compression requirement, a plurality of key frame sequences are selected and defined as the main key frame sequence.

[0078] It should be understood that different spatial position information corresponds to different shielding models, and the corresponding first shielding model can be selected based on the position information of the key frame sequence for shielding processing. For example, a face region corresponds to a shielding model, and a finger region corresponds to a shielding model.

[0079] Optionally, the number of the main key frame sequence can be one or more, and each main key frame sequence is a key frame sequence corresponding to a haptic signal.

[0080] Optionally, the first shielding processing of the secondary key frame according to the first shielding model can include the following process:

[0081] According to the shielding threshold and the number of shielding points, the key frames in the secondary key frame sequence whose distance range corresponding to the number of shielding points of the key frames in the main key frame sequence is less than the shielding threshold are deleted.

[0082] Optionally, the target key frame sequence can be understood as a masked key frame sequence, and binary encoding can be used to encode the target key frame sequence. The encoding length can be set according to actual needs, for example, the encoding length can be 8, 16 or 32.

[0083] Optionally, in some embodiments, performing a first masking process on the secondary key frame sequence according to the first masking model to obtain a target key frame sequence includes:

[0084] Inputting the frequency of the main key frame sequence into the first masking model to obtain a first masking threshold coefficient;

[0085] Multiplying the amplitude of the main key frame sequence by a first masking threshold coefficient to obtain a first masking threshold matrix;

[0086] A first masking process is performed on the secondary key frame sequence based on the first masking threshold matrix to obtain a target key frame sequence.

[0087] In an embodiment of the present application, the first masking processing performed on the sub-key frame sequence based on the first masking threshold matrix can be based on comparing the amplitude of each key frame in the sub-key frame sequence with the masking threshold of the key frame at the corresponding position in the first masking threshold matrix. When the amplitude of each key frame is less than the masking threshold of the key frame at the corresponding position in the first masking threshold matrix, the amplitude of the key frame is set to 0 or the sub-key frame is deleted (in this case, the key frame can be understood as a redundant signal). In other words, the first masking processing includes: when the amplitude of a target sub-key frame in the sub-key frame sequence is less than or equal to the masking threshold corresponding to the target sub-key frame in the first masking threshold matrix, the amplitude of the target sub-key frame is set to 0 or the target sub-key frame is deleted. Among them, the masking threshold of a position in a first masking threshold matrix corresponds to an interval range position. For example, the masking threshold of position 300 (assuming that the 300th signal frame is the primary key frame, position 300 is the position of the 300th signal frame in the first masking threshold matrix) corresponds to position 300 (the 301st signal frame in the secondary signal sequence) to position 328 (the 301st signal frame in the secondary signal sequence), that is, the signal frames corresponding to positions 301 to 328 contain the masking threshold of the secondary key frame corresponding to position 300.

[0088] Optionally, in some embodiments, performing a second masking process on the first signal based on a second masking model, and encoding the signal after the second masking process to obtain a second coded sequence includes:

[0089] Performing block processing on each of the first signals to obtain at least one signal block, where each signal block corresponds to a time interval;

[0090] For each signal block in the time interval, determining a primary signal block and a secondary signal block;

[0091] determining the second masking model according to the spatial modality mask;

[0092] performing a second masking process on the secondary signal block based on the second masking model to obtain a target signal block corresponding to each time interval, the target signal block including the primary signal block and the masked secondary signal block;

[0093] splicing the target signal blocks corresponding to all time intervals according to the correspondence between the signal blocks and the tactile signals to obtain multiple signals to be encoded;

[0094] performing encoding processing on the multiple signals to be encoded to obtain the second encoding sequence;

[0095] The spatial modality mask is determined based on the relative positions of the primary signal block and the secondary signal block.

[0096] In the embodiment of the present application, each tactile signal corresponds to a first signal, and the specific block length can be set according to actual needs. The length of the block will not directly affect the compression effect of the tactile signal, but the length of the block will affect the second masking process, thereby indirectly affecting the compression effect. Therefore, the block length can be understood as a hyperparameter and can be set according to actual conditions such as the processing performance and compression effect requirements of the tactile processing device. For example, in some embodiments, the block length can be 32, 64, 128 or 512, etc. It should be noted that, under the premise of meeting the compression fidelity and device processing capacity requirements, the length can be selected as a larger value as possible to improve the compression rate of the tactile signal.

[0097] Optionally, when performing block processing to obtain at least two signal blocks, each signal in each signal block is a partial signal of one first signal. For example, assuming that the first signal includes 1024 frames of signal and the length of the block is 512, the block processing includes two signal blocks, where the first signal block includes the 1st frame signal to the 512th frame signal, and the second signal block includes the 513th frame signal to the 1024th frame signal.

[0098] It should be understood that the first signals of different paths are processed in the same time interval, that is, one signal block of each path first signal is included in a certain time interval. Alternatively, in the embodiments of the present application, the main signal block in each time interval can be determined according to a predetermined algorithm, and the number of main signal blocks corresponding to one time interval can be one or more, which is not limited further herein. Among them, the main signal block should be the signal with larger amplitude or other strength indicators or the signal which is more sensitive to human body. Due to the existence of the main signal in the main signal block and the hysteresis of human tactile perception, the main signal in the main signal block has a spatial masking effect on other relatively weak secondary signals in the secondary signal block in a certain time period. The spatial masking effect makes it difficult for the human body to perceive the masked secondary signals through tactile perception, and the existence of these secondary signals has little significance for reproducing the tactile signals that the human body can perceive, which can be referred to as redundant tactile information. Removing redundant tactile information has little effect on the fidelity of the signal, and removing a large amount of redundant information can significantly improve the compression performance.

[0099] Alternatively, in the embodiments of the present application, the signals in the main signal block can be referred to as main signals, and the signals in the secondary signal block can be referred to as secondary signals.

[0100] Alternatively, the above predetermined algorithm can be set according to actual needs, and some possible schemes are described below:

[0101] 1. When the signal strength is approximately the same, determine the main signal block according to whether the position of the tactile signal is a tactile sensitive point. The input signal (i.e. the signal block of the first signal) corresponding to the tactile signal of the tactile sensitive point is taken as the main signal block. Among them, the tactile sensitive point can be obtained by subjective test, or defined according to specific equipment and specific parts.

[0102] 2. When there is a large interpolation of signal strength, the primary and secondary signal blocks can be defined according to the strength of the signal. For all input signals (i.e. signal blocks of the first signal), integrate the signals of each block according to factors such as amplitude and power, and sort the integral results in descending order. According to the compression requirement, select a number of signal blocks to define as main signal blocks.

[0103] Alternatively, it is usually difficult to test the masking effect of the main signal block at all frequencies by subjective test, and only the main signal block of a number of control frequency values is selected for spatial masking effect test. Therefore, in the case where the space-time joint masking effect model is obtained by subjective test, the second masking threshold coefficient of the main signal block masking effect at any frequency needs to be obtained by an interpolation function.

[0104] It should be noted that in the embodiments of the present application, the second shielding processing procedure (which includes: first determining the main signal block and the secondary signal block, then determining the second shielding model, and finally performing the second shielding processing) can be performed for each time interval, and then the above-mentioned second shielding processing procedure is executed through a loop to obtain the target signal block corresponding to each time interval.

[0105] Optionally, in some embodiments, one of the main signal blocks includes K main signals, K is a positive integer, and the second shielding processing of the secondary signal block based on the second shielding model to obtain the target signal block corresponding to each time interval includes:

[0106] Input the characteristic frequency of the main signal block into the second shielding model to obtain a second shielding threshold coefficient;

[0107] Multiply the amplitude of the main signal block by the second shielding threshold coefficient to obtain a second shielding threshold matrix;

[0108] Perform assignment processing on the second shielding threshold matrix based on the K main signals to obtain a third shielding threshold matrix;

[0109] Perform the second shielding processing of the secondary signal block based on the third shielding threshold matrix to obtain the target signal block;

[0110] Wherein, the assignment processing satisfies: in the case that the K main signals include at least one main signal that has not been processed, the assignment processing is performed; when the assignment processing is performed for the Pth time, the shielding threshold of the continuous n second positions after the first position in the second shielding threshold matrix is assigned as the shielding threshold of the first position, the first position is adjacent to the n second positions, the first position is the position corresponding to the main signal with the largest amplitude among the main signals that have not been processed in the previous P-1 times of assignment processing, and the main signals corresponding to the first position and the second position are the main signals processed in the Pth time of assignment processing, and n and P are positive integers.

[0111] In the embodiments of the present application, each main signal block corresponds to a second masking threshold matrix. When there are multiple main signal blocks, the same processing is performed for each main signal block, which will not be described again. Taking one main signal block as an example, in the first assignment processing, the K main signals can be sorted first to obtain the main signal with the largest amplitude in the K main signals, and the first position of the main signal with the largest amplitude in the K main signals in the second masking threshold matrix is determined. Then, the masking threshold values of the continuous n second positions after the first position in the second masking threshold matrix are assigned as the masking threshold value of the first position. Then, the main signal corresponding to the first position and the main signals corresponding to the continuous n second positions are marked as processed main signals. In the second assignment processing, the Q main signals (i.e., the main signals in the K main signals that have not been processed) can be sorted first to obtain the main signal with the largest amplitude in the Q main signals, and the first position of the main signal with the largest amplitude in the Q main signals in the second masking threshold matrix is determined. Then, the masking threshold values of the continuous n second positions after the first position in the second masking threshold matrix are assigned as the masking threshold value of the first position. Then, the main signal corresponding to the first position and the main signals corresponding to the continuous n second positions are marked as processed main signals. This cycle is executed until all main signals are processed.

[0112] Optionally, the unprocessed main signal can be understood as the main signal in the K main signals other than the main signal corresponding to the first position and the main signal corresponding to the second position.

[0113] For example, taking the block length equal to 1024 and n = 128 as an example, assuming that the position corresponding to the main signal with the largest amplitude is determined to be 300 in the first assignment processing, the positions 301 to 428 in the second masking threshold matrix can be assigned based on the masking threshold value of position 300. In the second assignment processing, the main signal with the largest amplitude is found in the main signals corresponding to all positions other than positions 300 to 428, and then the assignment is continued for the corresponding positions in the second masking threshold matrix. For example, assuming that the position corresponding to the main signal with the largest amplitude is determined to be 200 in the second assignment processing, the positions 201 to 328 in the second masking threshold matrix can be assigned based on the masking threshold value of position 200.

[0114] It should be noted that in some embodiments, the position corresponding to the processed primary signal can also be excluded from the assignment, for example, in the second assignment processing of the above embodiment, the positions 201 to 299 in the second masking threshold matrix can be assigned based on the masking threshold of position 200. It should be understood that the masking threshold of position 300 in the first assignment processing is greater than the masking threshold of position 200 in the second assignment processing (because the original primary signal corresponding to position 300 is greater than the original primary signal corresponding to position 200, and multiplying the same threshold coefficient still results in a masking threshold of position 300 greater than the masking threshold of position 200), so the assignment of positions 300 to 328 in the second assignment processing is "covered" by the greater assignment in the first assignment processing.

[0115] Optionally, in some embodiments, the second masking processing includes at least one of the following:

[0116] In the case that the amplitude of the first signal in the Lth secondary signal block is 0, the amplitude of the first signal is kept unchanged, and a first flag bit is added in the metadata, the first flag bit being used to indicate that the first signal does not need to be restored by masking;

[0117] In the case that the amplitude of the second signal in the Lth secondary signal block is not 0, the amplitude of the second signal is set to 0, and a second flag bit is added in the metadata, the second flag bit being used to indicate that the second signal needs to be restored by masking;

[0118] Wherein, L is a positive integer.

[0119] It should be noted that the decoding end can determine the second signal that needs to be restored based on the second flag bit (the amplitude of the second signal obtained by the decoding end is 0), and replace the amplitude of the second signal with the primary signal of M, thereby obtaining the second signal restored by masking. For example, the amplitude of a certain secondary signal in the Lth secondary signal block obtained by the decoding end is 0, and it is determined based on the second flag that the secondary signal needs to be restored by masking, assuming that the second masking threshold coefficient corresponding to the secondary signal is 0.5, and the corresponding proportional coefficient is 0.8, then in the decoding end, the secondary signal in the Lth secondary signal block can be replaced with a primary signal waveform with an amplitude of 0.5*0.8=0.4 times the amplitude of the primary signal.

[0120] Optionally, the rules for determining the proportional coefficient can include but are not limited to any one of the following:

[0121] 1. The same proportional coefficient is used for all secondary signal blocks, which is determined according to the ratio between the average amplitude of the masked secondary signal and the average amplitude of the corresponding primary signal;

[0122] 2. Different scaling factors are used for each signal block, which are determined according to the ratio between the average amplitude of the secondary signal and the average amplitude of the corresponding primary signal block.

[0123] Optionally, after obtaining the high-frequency signal blocks after shielding (i.e., the target signal blocks described above), signal combination can be performed to obtain new multi-channel high-frequency tactile signals (i.e., the multi-channel signals obtained after splicing). At this time, the multi-channel high-frequency tactile signals are signals after redundancy removal according to the spatial shielding effect and the time lagging principle of the shielding effect. Compared with the first signal, the multi-channel high-frequency tactile signals have reduced a large amount of tactile signals that cannot or are difficult to be perceived by the human body due to the spatial shielding effect, and basically ensure that the human body produces tactile perception difference for the processed signals.

[0124] Optionally, in some embodiments, the method further comprises:

[0125] performing a Fast Fourier Transform (FFT) operation on the primary signal block to obtain a frequency spectrum of the primary signal block;

[0126] determining the frequency component with the largest amplitude in the frequency spectrum of the primary signal block as the characteristic frequency of the primary signal block.

[0127] Optionally, in some embodiments, the method further comprises:

[0128] sending metadata (Metadata) to the decoding end, the metadata including at least one of:

[0129] the position number of the primary signal block and the spatial modality mask corresponding to the primary signal block;

[0130] a model identifier of a second shielding model used for performing a second shielding process on the first signal;

[0131] a flag bit indicating whether the signal with an amplitude of 0 in the secondary signal block needs to be restored by shielding;

[0132] a scaling factor for restoring the signal in the secondary signal block by shielding.

[0133] In the embodiments of the present application, the metadata described above is data describing data, mainly information describing data attributes, which is indispensable in signal encoding and decoding. Metadata also needs to be transmitted in vibration haptic signal encoding and decoding to facilitate the decoding end to restore the original vibration haptic signal.

[0134] Optionally, in some embodiments, the Metadata can further include information of the body region to which each signal belongs, position information within the body region, and relative position information between signals. If the haptic signal is a vibration haptic signal, the Metadata can further include sampling rate of the vibration haptic signal and other information.

[0135] For better understanding of the present application, the following refers to Figure 2 The encoding process of the present application is described. As shown in Figure 2 The encoding process of each haptic signal can include the following steps:

[0136] 21. Low-pass filtering the haptic signal to obtain a second signal (low-frequency signal component) and a third signal (high-frequency signal component);

[0137] 22. Extracting key frames from the second signal to obtain a key frame sequence, performing key frame masking (i.e., first masking process) on the key frame sequence to obtain a target key frame sequence, and then performing lossless binary encoding on the target key frame sequence to obtain a first encoded sequence;

[0138] 23. Reconstructing the low-frequency signal based on the key frame sequence, calculating the residual between the reconstructed low-frequency signal and the original second signal to obtain a target residual signal, and then superimposing the target residual signal into the third signal to obtain a first signal (high-frequency signal with low-frequency residual);

[0139] 24. Performing block processing on the first signal, determining a main signal block based on an integrator, determining a characteristic frequency of the main signal block, and determining a spatio-temporal masking effect model based on the characteristic frequency.

[0140] 25. Performing spatio-temporal masking on a secondary signal block of the first signal using the spatio-temporal masking effect model, and then performing wavelet transform and binary compression encoding.

[0141] 26. Combining the encoded sequences of the multiple signals after compression encoding to obtain a final encoded sequence.

[0142] Referring to Figure 3 , the present application further provides a haptic signal decoding method applied to a decoding end, as shown in Figure 3 The haptic signal decoding method includes:

[0143] Step 301, decoding the second encoded sequence to obtain multiple fourth signals;

[0144] Step 302, performing block processing on each of the fourth signals to obtain at least one signal block, each signal block corresponding to a time interval;

[0145] Step 303, determining the primary signal block and the secondary signal block corresponding to each time interval;

[0146] Step 304, for each time interval, performing a masking restoration on the secondary signal block based on the primary signal block to obtain a fifth signal, the fifth signal including the primary signal block and the masking-restored secondary signal block;

[0147] Step 305, obtaining a multi-channel decoded signal based on the fifth signal;

[0148] The second encoding sequence is obtained based on a first signal corresponding to each of the multi-channel haptic signals, the first signal being obtained based on a target residual signal and a third signal, the target residual signal being obtained based on a second signal, the second signal and the third signal being obtained based on a low-pass filtering processing on the haptic signal, and the frequency of the second signal being less than the frequency of the third signal.

[0149] Optionally, in some embodiments, the method further includes:

[0150] decoding the first encoding sequence to obtain a key frame sequence, the first encoding sequence being obtained based on the second signal;

[0151] performing an interpolation processing on the key frame sequence to obtain a sixth signal;

[0152] The obtaining of the multi-channel decoded signal based on the fifth signal includes performing a superposition processing on the fifth signal and the sixth signal to obtain the multi-channel decoded signal.

[0153] In the embodiments of the present application, the fourth signal can be understood as a high-frequency signal component with a low-frequency residual. Specifically, the decoding of the second encoding sequence includes binary decoding and inverse wavelet transform of the second encoding sequence, so as to obtain the fourth signal.

[0154] Optionally, the superposition processing can include an addition operation.

[0155] Optionally, in some embodiments, the obtaining of the fifth signal based on the primary signal block for each time interval includes:

[0156] For each time interval, performing a masking restoration on the secondary signal block based on a second masking threshold coefficient corresponding to the secondary signal block, a proportional coefficient corresponding to the secondary signal block, and an amplitude of the primary signal block to obtain the fifth signal.

[0157] Optionally, in some embodiments, the masking restoration includes:

[0158] replace the second secondary signal in the secondary signal block which needs to be restored by shielding with M times of target primary signals, the target primary signals being the primary signals in the primary signal block used for shielding the second secondary signal, wherein M is determined based on the second shielding threshold coefficient corresponding to the second secondary signal and the proportion coefficient corresponding to the second secondary signal.

[0159] In the embodiments of the present application, the value range of M is 0 to 1.

[0160] Optionally, in some embodiments, the method further comprises:

[0161] receiving metadata, the metadata comprising at least one of:

[0162] the position number of the primary signal block and the spatial modal mask corresponding to the primary signal block;

[0163] the model identification of the second shielding model used for the second shielding processing of the first signal;

[0164] a flag bit indicating whether the signal in the secondary signal block with an amplitude of 0 needs to be restored by shielding;

[0165] a proportion coefficient of the signal in the secondary signal block which needs to be restored by shielding.

[0166] In the embodiments of the present application, it can be determined whether the restoration is needed (i.e. whether the restoration by shielding is needed) according to the flag bit in the metadata, and when the restoration is needed, the corresponding secondary signal block can be restored by shielding according to the corresponding second shielding threshold coefficient, the corresponding proportion coefficient and the amplitude of the primary signal block.

[0167] Optionally, in some embodiments, the Metadata can further comprise the body region information to which each signal belongs, the position information in the body region, and the relative position information between each signal. If the above tactile signal is a vibration tactile signal, the Metadata can further comprise the sampling rate of the vibration tactile signal and other information.

[0168] It should be noted that after obtaining the above multi-channel decoded signals, the multi-channel decoded signals are matched with the body region and the position in the body region according to the position information in the Metadata. The primary signal and the secondary signal should also be matched with the corresponding positions respectively. The primary signal is very crucial for reproducing the vibration tactile signal, and the reproduction of the primary signal should be ensured first. The matching process can specifically include the following steps:

[0169] 1. Matching the body region to which each signal belongs.

[0170] 2. Matching the position of the signals belonging to the same body region in the body region.

[0171] 3. Add a priority mark to each signal belonging to the main signal. The signal added with the priority mark is processed preferentially in the subsequent step, ensuring the quality of the vibration haptic signal reproduction.

[0172] After obtaining the matching result, each channel signal is transmitted to the vibration haptic reproduction device at the position of the human body corresponding to the signal based on the matching result. Finally, each vibration haptic reproduction device receives the signal corresponding to the position where it is located from the decoder. After the device checks whether the signal matches the position where it is located, the vibration haptic signal reproduction is completed.

[0173] For better understanding of the present application, the following refers to Figure 4 The decoding process in the present application is described. As shown in Figure 4 The decoding process flow for each channel haptic signal can include the following steps:

[0174] 41. Perform signal blocking to obtain the first encoding sequence and the second encoding sequence corresponding to each channel haptic signal;

[0175] 42. Perform lossless binary decoding on the first encoding sequence to obtain a key frame sequence, and perform interpolation processing on the key frame sequence to obtain a sixth signal (i.e. a low frequency signal component);

[0176] 43. Perform binary compression decoding, wavelet transform and masking signal restoration processing on the second encoding sequence to obtain a fifth signal;

[0177] 44. Add the fourth signal and the fifth signal to obtain a channel decoded signal;

[0178] 45. Perform signal combination on the multiple channel decoded signals to obtain the final decoded signal.

[0179] The haptic signal encoding processing method provided in the embodiments of the present application can be executed by a haptic signal encoding processing device. In the embodiments of the present application, the haptic signal encoding processing method is executed by a haptic signal encoding processing device, which is taken as an example to describe the haptic signal encoding processing device provided in the embodiments of the present application.

[0180] Referring to Figure 5 , the embodiments of the present application also provide a haptic signal encoding processing device applied to the encoding end, as shown in Figure 5 The haptic signal encoding processing device 500 includes:

[0181] The acquisition module 501 is configured to acquire at least one of a key frame sequence corresponding to each of a plurality of haptic signals and a first signal, wherein the key frame sequence is obtained based on a second signal, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on the second signal and the key frame sequence, the second signal and the third signal are obtained based on low-pass filtering processing on the haptic signal, and the frequency of the second signal is less than the frequency of the third signal.

[0182] The execution module 502 is configured to perform a target operation, and the target operation includes at least one of the following:

[0183] performing first masking processing on the key frame sequence based on a first masking model, and encoding a signal after the first masking processing to obtain a first encoding sequence;

[0184] performing second masking processing on the first signal based on a second masking model, and encoding a signal after the second masking processing to obtain a second encoding sequence.

[0185] Optionally, the execution module 502 is specifically configured to perform the following operations:

[0186] determining a main key frame sequence and a secondary key frame sequence in the key frame sequence corresponding to the plurality of haptic signals;

[0187] determining the first masking model according to spatial position information corresponding to the main key frame sequence and spatial position information corresponding to the secondary key frame sequence;

[0188] performing first masking processing on the secondary key frame sequence according to the first masking model to obtain a target key frame sequence, and the target key frame sequence includes the main key frame sequence and the masked secondary key frame sequence;

[0189] encoding the target key frame sequence to obtain the first encoding sequence.

[0190] Optionally, the execution module 502 is specifically configured to perform the following operations:

[0191] inputting a frequency of the main key frame sequence into the first masking model to obtain a first masking threshold coefficient;

[0192] multiplying an amplitude of a reconstructed signal obtained by reconstructing the main key frame sequence with the first masking threshold coefficient to obtain a first masking threshold matrix;

[0193] performing first masking processing on the secondary key frame sequence based on the first masking threshold matrix to obtain the target key frame sequence.

[0194] Optionally, the main keyframe sequence satisfies at least one of the following:

[0195] The main keyframe sequence is a keyframe sequence corresponding to a haptic signal of a haptic sensitive point.

[0196] The main keyframe sequence is a keyframe sequence corresponding to a haptic signal with a signal strength greater than or equal to a preset strength threshold.

[0197] Optionally, the execution module 502 is specifically configured to perform the following operations:

[0198] The first signals are processed in blocks to obtain at least one signal block, each signal block corresponding to a time interval;

[0199] For the signal blocks in each time interval, a main signal block and a secondary signal block are determined;

[0200] The second masking model is determined according to the spatial modal mask;

[0201] The secondary signal block is subjected to second masking processing based on the second masking model to obtain a target signal block corresponding to each time interval, the target signal block including the main signal block and the masked secondary signal block;

[0202] The target signal blocks corresponding to all time intervals are spliced according to the correspondence between the signal blocks and the haptic signals to obtain a plurality of to-be-encoded signals;

[0203] The plurality of to-be-encoded signals are subjected to encoding processing to obtain the second encoding sequence;

[0204] The spatial modal mask is determined based on the relative positions of the main signal block and the secondary signal block.

[0205] Optionally, one main signal block includes K main signals, K being a positive integer, and the execution module 502 is specifically configured to perform the following operations:

[0206] The characteristic frequency of the main signal block is input into the second masking model to obtain a second masking threshold coefficient;

[0207] The amplitude of the main signal block is multiplied by the second masking threshold coefficient to obtain a second masking threshold matrix;

[0208] The K main signals are subjected to assignment processing based on the second masking threshold matrix to obtain a third masking threshold matrix;

[0209] The secondary signal block is subjected to second masking processing based on the third masking threshold matrix to obtain the target signal block;

[0210] The assignment processing satisfies: in a case where the K main signals include at least one main signal that is not processed, the assignment processing is performed; in a case where the assignment processing is performed for a Pth time, a masking threshold value in a first position after n second positions in the second masking threshold matrix is assigned to a masking threshold value in the first position, the first position is adjacent to the n second positions, the first position is a position corresponding to a main signal with a maximum amplitude among the main signals that are not processed in a previous P-1th assignment processing, and a main signal corresponding to the first position and the second position is a main signal that is processed in the Pth assignment processing, n and P are positive integers.

[0211] Optionally, the second masking processing includes at least one of the following:

[0212] In a case where an amplitude of a first signal in an Lth signal block is 0, the amplitude of the first signal is kept unchanged, and a first flag bit is added in metadata, the first flag bit being used to indicate that the first signal does not need to be subjected to masking restoration;

[0213] In a case where an amplitude of a second signal in the Lth signal block is not 0, the amplitude of the second signal is set to 0, and a second flag bit is added in the metadata, the second flag bit being used to indicate that the second signal needs to be subjected to masking restoration;

[0214] wherein L is a positive integer.

[0215] Optionally, the haptic signal encoding processing apparatus 500 further includes:

[0216] a transformation module, configured to perform a fast Fourier transform (FFT) operation on the main signal block to obtain a frequency spectrum of the main signal block;

[0217] a second determination module, configured to determine a frequency component with a maximum amplitude in the frequency spectrum of the main signal block as a characteristic frequency of the main signal block.

[0218] Optionally, the haptic signal encoding processing apparatus 500 further includes:

[0219] a sending module, configured to send metadata to a decoding end, the metadata including at least one of the following:

[0220] a position number of the main signal block and a spatial modality mask corresponding to the main signal block;

[0221] a model identifier of a second masking model used to perform second masking processing on the first signal;

[0222] a flag bit indicating whether a signal with an amplitude of 0 in the signal block needs to be subjected to masking restoration;

[0223] A proportion coefficient of the signals in the signal block that need to be shielded and restored.

[0224] The execution subject of the haptic signal decoding processing method provided in the embodiments of the present application can be a haptic signal decoding processing device. The haptic signal decoding processing device provided in the embodiments of the present application is taken as an example to illustrate the haptic signal decoding processing device provided in the embodiments of the present application.

[0225] With reference to Figure 6 , the embodiments of the present application further provide a haptic signal decoding processing device applied to a decoding end, as shown in Figure 6 The haptic signal decoding processing device 600 includes:

[0226] A decoding module 601 is configured to decode a second encoding sequence to obtain a plurality of fourth signals.

[0227] A block processing module 602 is configured to perform block processing on each of the fourth signals to obtain at least one signal block, each signal block corresponding to a time interval.

[0228] A determination module 603 is configured to determine a main signal block and a secondary signal block corresponding to each time interval.

[0229] A restoration module 604 is configured to, for each time interval, perform shielded restoration on the secondary signal block based on the main signal block to obtain a fifth signal, the fifth signal including the main signal block and the shielded and restored secondary signal block.

[0230] A processing module 605 is configured to obtain a plurality of decoding signals based on the fifth signal.

[0231] The second encoding sequence is obtained based on a first signal corresponding to each of the plurality of haptic signals, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on a second signal, and the second signal and the third signal are obtained based on low-pass filtering processing on the haptic signals, and the frequency of the second signal is less than the frequency of the third signal.

[0232] Optionally, the restoration module 604 is specifically configured to, for each time interval, perform shielded restoration on the secondary signal block according to a second shield threshold coefficient corresponding to the secondary signal block, a proportion coefficient corresponding to the secondary signal block, and an amplitude of the main signal block to obtain the fifth signal.

[0233] Optionally, the shielded restoration includes:

[0234] Replace the second secondary signal in the secondary signal block which needs to be shielded and restored with M times of the target primary signal, the target primary signal being a primary signal in the primary signal block used for shielding the second secondary signal, wherein M is determined based on the second shielding threshold coefficient corresponding to the second secondary signal and the proportion coefficient corresponding to the second secondary signal.

[0235] Optionally, the haptic signal decoding processing apparatus 600 further comprises:

[0236] The receiving module 601 is configured to receive metadata, and the metadata comprises at least one of the following:

[0237] The position number of the primary signal block and the spatial modal mask corresponding to the primary signal block;

[0238] The model identifier of the second shielding model used for performing the second shielding processing on the first signal;

[0239] The flag bit indicating whether the signal with an amplitude of 0 in the secondary signal block needs to be shielded and restored;

[0240] The proportion coefficient of the signal in the secondary signal block which needs to be shielded and restored.

[0241] Optionally, the decoding module 601 is further configured to decode a first encoding sequence to obtain a key frame sequence, the first encoding sequence being obtained based on the second signal; and perform interpolation processing on the key frame sequence to obtain a sixth signal.

[0242] The processing module 605 is specifically configured to perform superposition processing on the fifth signal and the sixth signal to obtain a multi-channel decoded signal.

[0243] The haptic signal encoding processing apparatus or the haptic signal decoding processing apparatus in the embodiments of the present application can be an electronic device, for example, an electronic device with an operating system, or a component in an electronic device, for example, an integrated circuit or a chip. The electronic device can be a terminal or other devices except the terminal. Exemplarily, the terminal can include but is not limited to the types of the terminal 11 listed above, and the other devices can be a server, a network attached storage (NAS) or the like, which are not limited in the embodiments of the present application.

[0244] The haptic signal encoding processing apparatus and the haptic signal decoding processing apparatus provided in the embodiments of the present application can implement each process of the method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein. Figures 1 to 4

[0245] As Figure 7 ​As shown, the embodiments of the present application further provide a communication device 700, comprising a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions executable on the processor 701. For example, when the communication device 700 is an encoding end device, the programs or instructions are executed by the processor 701 to implement the steps of the above-mentioned haptic signal encoding processing method embodiments, and achieve the same technical effects. When the communication device 700 is a decoding end device, the programs or instructions are executed by the processor 701 to implement the steps of the above-mentioned haptic signal decoding processing method embodiments, and achieve the same technical effects. To avoid repetition, details are not described herein. Optionally, the memory can further store a space-time joint masking effect model.

[0246] The embodiments of the present application further provide an electronic device, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to run programs or instructions to implement the steps in the above-mentioned method embodiments. The electronic device embodiments correspond to the above-mentioned method embodiments, and each implementation process and implementation manner of the above-mentioned method embodiments can be applied to the electronic device embodiments, and achieve the same technical effects. Specifically, Figures 1 to 4 The embodiments of the present application further provide an electronic device, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to run programs or instructions to implement the steps in the above-mentioned method embodiments. The electronic device embodiments correspond to the above-mentioned method embodiments, and each implementation process and implementation manner of the above-mentioned method embodiments can be applied to the electronic device embodiments, and achieve the same technical effects. Specifically, Figure 8 A hardware structure schematic diagram of an electronic device for implementing the embodiments of the present application.

[0247] The electronic device 800 includes, but is not limited to, at least part of the following components: a radio frequency unit 801, a network module 802, an output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810, etc.

[0248] Those skilled in the art can understand that the electronic device 800 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 810 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the above-mentioned embodiments does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than those shown, or combine certain components, or have different component arrangements, which are not described herein.

[0249] It should be understood that in the embodiments of the present application, the input unit 804 can include a tactile signal collection sensor and a sensor host computer for collecting, processing and converting sensor information. The display unit 806 can include a display panel 8061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 can include two parts of a touch detection device and a touch controller. The other input devices 8072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc., which will not be described here.

[0250] In the embodiments of the present application, after the radio frequency unit 801 receives the downlink data from the network side device, it can be transmitted to the processor 810 for processing. In addition, the radio frequency unit 801 can send uplink data to the network side device. Generally, the radio frequency unit 801 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0251] The memory 809 can be used to store software programs or instructions and various data (e.g., data that can include a spatiotemporal joint masking effect model and a haptic signal, etc.). The memory 809 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), etc. In addition, the memory 809 can include a volatile memory or a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 809 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0252] The processor 810 can include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 810.

[0253] The processor 810 is configured to obtain at least one of a key frame sequence corresponding to each of the plurality of haptic signals and a first signal when the electronic device is an encoding end, wherein the key frame sequence is obtained based on a second signal, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on the second signal and the key frame sequence, the second signal and the third signal are obtained based on low-pass filtering processing on the haptic signal, and a frequency of the second signal is less than a frequency of the third signal; perform a target operation; and wherein the target operation comprises at least one of performing first masking processing on the key frame sequence based on a first masking model and encoding the signal after the first masking processing to obtain a first encoding sequence, and performing second masking processing on the first signal based on a second masking model and encoding the signal after the second masking processing to obtain a second encoding sequence.

[0254] The processor 810 is configured to decode the second encoding sequence to obtain a plurality of fourth signals when the electronic device is a decoding end; perform block processing on each of the fourth signals to obtain at least one signal block, each signal block corresponding to a time interval; determine a main signal block and a secondary signal block corresponding to each time interval; perform masking restoration on the secondary signal block based on the main signal block for each time interval to obtain a fifth signal, the fifth signal comprising the main signal block and the secondary signal block after masking restoration; and obtain a plurality of decoding signals based on the fifth signal; wherein the second encoding sequence is obtained based on the first signal corresponding to each of the plurality of haptic signals, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on a second signal, the second signal and the third signal are obtained based on low-pass filtering processing on the haptic signal, and a frequency of the second signal is less than a frequency of the third signal.

[0255] It can be understood that the implementation processes of the implementation manners mentioned in the embodiment can refer to the related descriptions of the above method embodiments and achieve the same or corresponding technical effects. To avoid repetition, they will not be described here again.

[0256] The embodiment of the present application further provides a readable storage medium, the readable storage medium stores a program or instructions, the program or instructions are executed by a processor to implement each process of the above haptic signal encoding processing method embodiment or implement each process of the above haptic signal decoding processing method embodiment, and the same technical effects can be achieved. To avoid repetition, they will not be described here again. Optionally, the above readable storage medium can further store a spatio-temporal joint masking effect model.

[0257] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0258] The chip provided by the embodiments of the present application includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used to run programs or instructions to realize the processes of the above-mentioned haptic signal encoding processing method embodiments or the processes of the above-mentioned haptic signal decoding processing method embodiments, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0259] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0260] The embodiments of the present application further provide a computer program / program product stored in a storage medium, which is executed by at least one processor to realize the processes of the above-mentioned haptic signal encoding processing method embodiments or the processes of the above-mentioned haptic signal decoding processing method embodiments, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0261] The embodiments of the present application further provide a haptic signal encoding and decoding system, including an encoding end device and a decoding end device, the terminal can be used to execute the steps of the above-mentioned haptic signal encoding processing method, and the network side device can be used to execute the steps of the above-mentioned haptic signal decoding processing method.

[0262] It should be noted that in this paper, the term "including", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiments of the present application is not limited to the order of functions shown or discussed, but also includes the functions performed in a substantially simultaneous manner or in the opposite order, for example, the described method can be performed in a different order from the described order, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0263] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of a computer software product and a general hardware platform as necessary, and of course can also be realized by hardware. The computer software product is stored in a storage medium (such as a ROM, a RAM, a magnetic disc, an optical disc, etc.), and includes a plurality of instructions for enabling a terminal or a network side device to execute the method described in each embodiment of the present application.

[0264] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are only illustrative but not restrictive. Those skilled in the art can make many forms of embodiments under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims, and these embodiments all belong to the protection of the present application.

Claims

1. A tactile signal encoding and processing method, applied to an encoding end, characterized in that: include: Obtaining at least one of a key frame sequence and a first signal corresponding to each tactile signal in multiple tactile signals, wherein the key frame sequence is obtained based on the second signal, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on the second signal and the key frame sequence, the second signal and the third signal are obtained by low-pass filtering the tactile signal, and a frequency of the second signal is lower than a frequency of the third signal; Execute a target operation; wherein the target operation includes at least one of the following: Performing a first masking process on the key frame sequence based on a first masking model, and encoding the signal after the first masking process to obtain a first coded sequence; A second masking process is performed on the first signal based on a second masking model, and the signal after the second masking process is encoded to obtain a second coding sequence.

2. The method according to claim 1, characterized in that The performing a first masking process on the key frame sequence based on the first masking model to obtain a first coding sequence includes: Determining a primary key frame sequence and a secondary key frame sequence in the key frame sequences corresponding to the multiple tactile signals; determining the first occlusion model according to spatial position information corresponding to the primary key frame sequence and spatial position information corresponding to the secondary key frame sequence; performing a first masking process on the secondary key frame sequence according to the first masking model to obtain a target key frame sequence, wherein the target key frame sequence includes the primary key frame sequence and the masked secondary key frame sequence; The target key frame sequence is encoded to obtain a first encoded sequence.

3. The method according to claim 2, characterized in that The performing a first masking process on the secondary key frame sequence according to the first masking model to obtain a target key frame sequence includes: Inputting the frequency of the main key frame sequence into the first masking model to obtain a first masking threshold coefficient; Multiplying the amplitude of the main key frame sequence by a first masking threshold coefficient to obtain a first masking threshold matrix; A first masking process is performed on the secondary key frame sequence based on the first masking threshold matrix to obtain a target key frame sequence.

4. The method according to claim 2, characterized in that The main key frame sequence satisfies at least one of the following: The main key frame sequence is a key frame sequence corresponding to the tactile signal of the tactile sensitive point; The main key frame sequence is a key frame sequence corresponding to a tactile signal having a signal strength greater than or equal to a preset strength threshold.

5. The method according to claim 1, characterized in that The performing a second masking process on the first signal based on a second masking model, and encoding the signal after the second masking process to obtain a second coded sequence includes: Performing block processing on each of the first signals to obtain at least one signal block, where each signal block corresponds to a time interval; For each signal block in the time interval, determining a primary signal block and a secondary signal block; determining the second masking model according to the spatial modality mask; performing a second masking process on the secondary signal block based on the second masking model to obtain a target signal block corresponding to each time interval, the target signal block including the primary signal block and the masked secondary signal block; splicing the target signal blocks corresponding to all time intervals according to the correspondence between the signal blocks and the tactile signals to obtain multiple signals to be encoded; performing encoding processing on the multiple signals to be encoded to obtain the second encoding sequence; The spatial modality mask is determined based on the relative positions of the primary signal block and the secondary signal block.

6. The method according to claim 5, characterized in that One of the primary signal blocks includes K primary signals, where K is a positive integer. The performing of the second masking process on the secondary signal block based on the second masking model to obtain a target signal block corresponding to each time interval includes: Inputting the characteristic frequency of the main signal block into the second masking model to obtain a second masking threshold coefficient; multiplying the amplitude of the main signal block by the second masking threshold coefficient to obtain a second masking threshold matrix; Performing assignment processing on the second masking threshold matrix based on the K main signals to obtain a third masking threshold matrix; performing a second masking process on the secondary signal block based on the third masking threshold matrix to obtain the target signal block; In which, the assignment processing satisfies: when the K main signals include at least one unprocessed main signal, the assignment processing is performed; when the assignment processing is performed for the Pth time, the masking thresholds of the consecutive n second positions located after the first position in the second masking threshold matrix are assigned as the masking thresholds of the first position, the first position is adjacent to the n second positions, the first position is the position corresponding to the main signal with the largest amplitude among the main signals that have not been processed when the assignment processing was performed for the previous P-1 times, and the main signals corresponding to the first position and the second position are the main signals processed when the assignment processing is performed for the Pth time, and n and P are both positive integers.

7. The method according to claim 6, characterized in that The second masking process includes at least one of the following: When the amplitude of the first signal in the Lth secondary signal block is 0, the amplitude of the first signal is kept unchanged, and a first flag is added to the metadata, where the first flag is used to indicate that the first signal does not need to be masked and restored; If the amplitude of the second signal in the Lth secondary signal block is not 0, the amplitude of the second signal is set to 0, and a second flag is added to the metadata, where the second flag is used to indicate that masking restoration needs to be performed on the second signal; Wherein, L is a positive integer.

8. The method according to claim 6, characterized in that The method further comprises: Performing a fast Fourier transform (FFT) operation on the main signal block to obtain a frequency spectrum of the main signal block; A frequency component with the largest amplitude in the frequency spectrum of the main signal block is determined as a characteristic frequency of the main signal block.

9. The method according to any one of claims 5 to 8, characterized in that The method further comprises: Send metadata to the decoding end, where the metadata includes at least one of the following: a position number of the main signal block and a spatial modality mask corresponding to the main signal block; a model identifier of a second masking model used to perform a second masking process on the first signal; A flag indicating whether the signal with an amplitude of 0 in the secondary signal block needs to be masked and restored; The proportional coefficient of the signal in the secondary signal block that needs to be masked and restored.

10. A tactile signal decoding and processing method, applied to a decoding end, characterized in that: include: Decoding the second coding sequence to obtain multiple fourth signals; Performing block processing on each of the fourth signals to obtain at least one signal block, where each signal block corresponds to a time interval; Determine the primary signal block and the secondary signal block corresponding to each time interval; For each of the time intervals, masking and restoring the secondary signal block based on the primary signal block to obtain a fifth signal, wherein the fifth signal includes the primary signal block and the masked and restored secondary signal block; Based on the fifth signal, obtaining a multi-channel decoded signal; Among them, the second coding sequence is obtained based on the first signal corresponding to each tactile signal in the multiple tactile signals, the first signal is obtained based on the target residual signal and the third signal, the target residual signal is obtained based on the second signal, the second signal and the third signal are obtained based on low-pass filtering of the tactile signal, and the frequency of the second signal is less than the frequency of the third signal.

11. The method according to claim 10, characterized in that For each of the time intervals, masking and restoring the secondary signal block based on the primary signal block to obtain the fifth signal includes: For each of the time intervals, the secondary signal block is masked and restored according to the second masking threshold coefficient corresponding to the secondary signal block, the proportional coefficient corresponding to the secondary signal block, and the amplitude of the primary signal block to obtain the fifth signal.

12. The method according to claim 11, characterized in that The mask restoration includes: The secondary signal that needs to be masked and restored in the secondary signal block is replaced with M times the target main signal, where the target main signal is the main signal in the main signal block used to mask the secondary signal, wherein M is determined based on the second masking threshold coefficient corresponding to the second signal and the proportional coefficient corresponding to the second signal.

13. The method according to any one of claims 10 to 12, characterized in that The method further comprises: Receive metadata, the metadata including at least one of the following: a position number of the main signal block and a spatial modality mask corresponding to the main signal block; a model identifier of a second masking model used to perform a second masking process on the first signal; A flag indicating whether the signal with an amplitude of 0 in the secondary signal block needs to be masked and restored; The proportional coefficient of the signal in the secondary signal block that needs to be masked and restored.

14. The method according to any one of claims 10 to 12, characterized in that The method further comprises: decoding a first coded sequence to obtain a key frame sequence, where the first coded sequence is obtained based on the second signal; performing interpolation processing on the key frame sequence to obtain a sixth signal; Wherein, obtaining a multi-channel decoded signal based on the fifth signal includes: performing superposition processing on the fifth signal and the sixth signal to obtain a multi-channel decoded signal.

15. A tactile signal encoding and processing device, applied to an encoding end, characterized in that: include: an acquisition module, configured to acquire at least one of a key frame sequence and a first signal corresponding to each tactile signal in multiple tactile signals, wherein the key frame sequence is obtained based on the second signal, the first signal is obtained based on a target residual signal and a third signal, the target residual signal is obtained based on the second signal and the key frame sequence, the second signal and the third signal are obtained by low-pass filtering the tactile signal, and a frequency of the second signal is lower than a frequency of the third signal; An execution module is configured to execute a target operation; wherein the target operation includes at least one of the following: Performing a first masking process on the key frame sequence based on a first masking model, and encoding the signal after the first masking process to obtain a first coded sequence; A second masking process is performed on the first signal based on a second masking model, and the signal after the second masking process is encoded to obtain a second coding sequence.

16. A tactile signal decoding and processing device, applied to a decoding end, characterized in that: include: A decoding module, configured to decode the second coding sequence to obtain multiple fourth signals; A block processing module, configured to perform block processing on each channel of the fourth signal to obtain at least one signal block, where each signal block corresponds to a time interval; A first determining module, configured to determine a primary signal block and a secondary signal block corresponding to each time interval; a restoration module, configured to perform masking restoration on the secondary signal block based on the primary signal block for each of the time intervals to obtain a fifth signal, wherein the fifth signal includes the primary signal block and the masked and restored secondary signal block; a processing module, configured to obtain a multi-channel decoded signal based on the fifth signal; Among them, the second coding sequence is obtained based on the first signal corresponding to each tactile signal in the multiple tactile signals, the first signal is obtained based on the target residual signal and the third signal, the target residual signal is obtained based on the second signal, the second signal and the third signal are obtained based on low-pass filtering of the tactile signal, and the frequency of the second signal is less than the frequency of the third signal.

17. An electronic device, characterized in that: The device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the tactile signal encoding and processing method according to any one of claims 1 to 9 are implemented, or when the program or instruction is executed by the processor, the steps of the tactile signal decoding and processing method according to any one of claims 10 to 14 are implemented.

18. A readable storage medium, characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the tactile signal encoding processing method according to any one of claims 1 to 9 are implemented, or the steps of the tactile signal decoding processing method according to any one of claims 10 to 14 are implemented.

Images