Media file processing method, device, and equipment

Abstract

The present disclosure provides a media file processing method, apparatus, and device, which belong to the technical field of media file transmission. The media file processing method according to an embodiment of the present disclosure includes the steps of: determining a grayscale image track and each haptic signal track corresponding to video data, the grayscale image track including a grayscale image corresponding to each frame screen in the video data, the grayscale image corresponding to each frame screen representing a position distribution of each haptic area in each frame screen, and the haptic signal track including a haptic signal corresponding to each haptic area in the video data (S101); generating a first mapping table based on the grayscale values ​​in the grayscale image track and the haptic signal track; and encapsulating the first mapping table, the grayscale image track, the haptic signal track, and the video data to obtain a target media file (S103).

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This disclosure claims priority to a Chinese patent application filed with the China Patent Office on February 27, 2023, bearing application number 202310198350.2 and entitled "Media File Processing Method, Apparatus, and Device," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the technical field of media file transmission, and more particularly to a media file processing method, apparatus, and device. [Background technology]

[0003] Haptic technology offers a new way of entertainment and sensory immersion beyond audio and video media. Take media streams such as streaming games or mobile advertisements as examples, adding haptics to the audio / video content of the media stream can greatly enhance the user's experience and enjoyment of the media content.

[0004] For this reason, haptics has been proposed as a primary media type on par with audio and video in the ISO Base Media File Format (ISOBMFF) and as a supplement to the Motion Picture Experts Group Dynamic Adaptive Streaming over HTTP (MPEG-DASH) standard, but this approach can only meet the needs of a single-image, single-haptic interaction setting. Summary of the Invention

[0005] In a first aspect, there is provided a method for processing a media file, said method comprising: determining a grayscale image track and each haptic signal track corresponding to video data, wherein the grayscale image track includes a grayscale image corresponding to each frame screen in the video data, the grayscale image corresponding to each frame screen representing a positional distribution of each tactile area in each frame screen, and the haptic signal track includes a haptic signal corresponding to each tactile area in the video data; generating a first mapping table based on each grayscale value in the grayscale image track and each haptic signal track; Encapsulating the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

[0006] In a second aspect, there is provided a method for processing a media file, said method comprising: acquiring and decapsulating the target media file to obtain video data, a first mapping table, a grayscale image track corresponding to the video data, and each haptic signal track, wherein the grayscale image track includes a grayscale image corresponding to each frame screen in the video data, and the grayscale image corresponding to each frame screen represents a position distribution of each haptic area in each frame screen; outputting the video data and mounting the grayscale image track; Mounting a haptic signal track corresponding to the grayscale image currently being output in the grayscale image track based on the grayscale image track and the first mapping table; outputting a haptic signal in a haptic signal track corresponding to the currently output grayscale image.

[0007] In a third aspect, there is provided a media file processing device, the device comprising: a first determination module that determines a grayscale image track and each haptic signal track corresponding to video data, wherein the grayscale image track includes grayscale images corresponding to each frame screen in the video data, the grayscale images corresponding to each frame screen representing a positional distribution of each tactile area in each frame screen, and the haptic signal tracks include haptic signals corresponding to each tactile area in the video data; a first generation module that generates a first mapping table based on each grayscale value in the grayscale image track and each haptic signal track; a first encapsulation module that encapsulates the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

[0008] In a fourth aspect, there is provided a media file processing device, the device comprising: a first processing module that acquires and decapsulates a target media file to obtain video data, a first mapping table, a grayscale image track corresponding to the video data, and each haptic signal track, wherein the grayscale image track includes a grayscale image corresponding to each frame screen in the video data, and the grayscale image corresponding to each frame screen represents a position distribution of each haptic area in each frame screen; a second processing module that outputs the video data and mounts the grayscale image track; a third processing module that mounts a haptic signal track corresponding to the grayscale image currently being output in the grayscale image track based on the grayscale image track and the first mapping table; and a fourth processing module that outputs a haptic signal in a haptic signal track that corresponds to the currently output grayscale image.

[0009] In a fifth aspect, there is provided a media file processing device comprising a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the media file processing method described in the first aspect.

[0010] In a sixth aspect, there is provided a media file processing device comprising a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the media file processing method described in the second aspect.

[0011] In a seventh aspect, a memory having computer readable code stored therein; one or more processors; There is provided a computing device that, when the computer readable code is executed by the one or more processors, performs the media file processing method according to the first or second aspect.

[0012] In an eighth aspect, there is provided a computer program comprising computer readable code which, when executed on a computing device, causes the computing device to perform the media file processing method according to the first or second aspect.

[0013] In a ninth aspect, there is provided a computer readable medium having stored thereon a computer program according to the eighth aspect.

[0014] In a tenth aspect, there is provided a media file processing system comprising: a video production device for performing the steps of the media file processing method set forth in the first aspect above; and a video output device for performing the steps of the media file processing method set forth in the second aspect above.

[0015] In an eleventh aspect, there is provided a readable storage medium having stored thereon a program or instructions which, when executed by a processor, implements the steps of the media file processing method described in the first aspect or the steps of the media file processing method described in the second aspect.

[0016] In a twelfth aspect, there is provided a chip comprising a processor and a communications interface, the communications interface being coupled to the processor, the processor executing a program or instructions to implement the media file processing method described in the first aspect or the media file processing method described in the second aspect.

[0017] In a thirteenth aspect, there is provided a computer program / program product stored on a storage medium and which, when executed by at least one processor, implements the steps of the media file processing method according to the first or second aspect.

[0018] The above description is merely a summary of the technical solutions of the present disclosure. In order to make the technical solutions of the present disclosure more clearly understood and implemented in accordance with the content of the specification, and to make the above and other objectives, features and advantages of the present disclosure more clearly comprehensible, specific embodiments of the present disclosure are given below.

[0019] In order to more clearly describe the technical solutions in the embodiments of the present disclosure or related technologies, the drawings necessary for describing the embodiments or related technologies will be briefly described below. The drawings in the following description are some embodiments of the present disclosure, and it is obvious to those skilled in the art that other drawings can be obtained based on these drawings without any creative efforts. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram of a conventional media file encapsulation principle in an embodiment of the present disclosure. [Figure 2] 1 is a flowchart of a media file processing method according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a schematic diagram of the processing flow of a player in an embodiment of the present disclosure. [Figure 4] FIG. 10 is a schematic diagram of starting samples of multiple haptic signal tracks in an embodiment of the present disclosure. [Figure 5] FIG. 1 is a schematic diagram of a grayscale image according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a schematic diagram of a flow for realizing remote surgery in an embodiment of the present disclosure. [Figure 7] 10 is a flowchart of another media file processing method according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a block diagram illustrating a configuration of a media file processing device according to an embodiment of the present disclosure. [Figure 9] FIG. 10 is a block diagram illustrating the configuration of another media file processing device according to an embodiment of the present disclosure. [Figure 10] 1 shows a schematic block diagram of a computing device for performing the methods of the present disclosure; [Figure 11] 1 illustrates a schematic representation of a storage unit for holding or carrying program code for implementing the methods according to the present disclosure; DETAILED DESCRIPTION OF THE INVENTION

[0021] In order to clarify the objectives, technical solutions, and advantages of the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. It is clear that the described embodiments are only a part of the embodiments of the present disclosure, and are not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without any creative efforts fall within the scope of protection of the present disclosure.

[0022] Terms such as "first," "second," and the like used in the specification and claims of the present disclosure are intended to distinguish between similar objects and are not intended to describe a particular order or priority. It should be understood that terms used in this manner may be interchanged, where appropriate, so that the embodiments of the present disclosure are implemented in an order other than that illustrated or described herein. Objects distinguished by "first" and "second" are typically of one type and do not limit the number of objects; for example, a first object may be one or at least two. Furthermore, the term "and / or" used in the specification and claims represents at least one of the connected objects, and the symbol " / " generally indicates that the related objects before and after are in an "or" relationship.

[0023] To better understand the media file processing method of the embodiments of the present disclosure, some related basic concepts are briefly introduced below.

[0024] Box: Also known as a frame, container, or atom. Box is defined as "An object-oriented building block defined by a unique type identifier and length (called 'atom' in some specifications, including the first definition of MP4)", which translates to "An object-oriented building block defined by a unique type identifier and length (called 'atom' in some specifications, including the first definition of MP4)". Container Box: Container Box is defined as "A box whose sole purpose is to contain and group a set of related boxes", which translates to "A box whose sole purpose is to store and group a set of related boxes". Movie Box: Movie Box is defined as "A container box whose sub-boxes define the metadata for a presentation ('moov')", which translates to "Also known as a moov box, this is a container box whose sub-boxes define the metadata ('moov') required for a media presentation". Media Data Box: Media Data Box is defined as "A container box which can hold the actual media data for a presentation ('mdat')", which translates to "A container box which can hold the actual media data ('mdat') for a media presentation".ISO Base Media File: ISO Base Media File is defined as "The name of the file format described in this specification", where a box is the basic unit that makes up an ISO Base Media File. Presentation: Presentation is defined as "One or more motion sequences (qv), possibly combined with audio", which is translated as "One or more motion sequences that may include audio". Sample: Sample is defined as "In non-hint tracks, a sample is an individual frame of video, a time-contiguous series of video frames, or a time-contiguous compressed section of audio. In hint tracks, a sample defines the formation of one or more streaming packets. No two samples within a track may share the same time-stamp," which can be translated as "In non-hint tracks, a sample is an individual frame of video, a time-contiguous series of video frames, or a time-contiguous compressed section of audio. In hint tracks, a sample defines the formation of one or more streaming packets. No two samples in a track may share the same time-stamp." A sample in a grayscale image track / haptic signal track described in this disclosure means a single grayscale image / haptic signal, or multiple time-contiguous grayscale images / haptic signals.Sample Description: Sample Description is defined as "A structure which defines and describes the format of some number of samples in a track," which is translated as "a structure that defines and describes the format of some samples in a track." Sample Table: Sample Table is defined as "A packed directory for the timing and physical layout of the samples in a track," which is translated as "a directory that packs the timing and physical layout of the samples in a track." Chunk: Chunk is defined as "A contiguous set of samples for one track," which is translated as "a contiguous set of samples for one track." Track: Track is defined as "A collection of related samples (qv) in an ISO base media file. For media data, a track corresponds to a sequence of images or sampled audio. For hint tracks, a track corresponds to a streaming channel," which is translated as "a collection of related samples in an ISO base media file. For media data, a track corresponds to a sequence of images or sampled audio. For hint tracks, a track corresponds to a streaming channel." That is, a track refers to a series of samples with time attributes encapsulated according to the ISO Base Media File Format.Hint Track: Hint Track is defined as "A special track which does not contain media data. Instead, it contains instructions for packaging one or more tracks into a streaming channel," which translates to "A special track which does not contain media data. Instead, it contains instructions for packaging one or more tracks into a streaming channel." Hinter: Hinter is defined as "A tool that is run on a file containing only media, to add one or more hint tracks to the file and so facilitate streaming," which translates to "A tool that is run on a file containing only media, to add one or more hint tracks to the file and so facilitate streaming."

[0025] Figure 1 illustrates the principle of conventional media file encapsulation. A video production device compresses video data, audio data, and haptic data into codestreams, then encapsulates them into a single encapsulated media file according to the ISO base media file format and transmits the encapsulated media file to a video playback device (i.e., a player). The player sequentially decapsulates and decompresses the encapsulated media file, triggering corresponding actuators to output the corresponding data. For example, it can trigger a display, speaker, and vibration motor to output the video data, audio data, and haptic data, respectively.

[0026] Although haptics has been proposed as a primary media type equivalent to audio and video in the ISO Base Media File Format and as a supplement to the MPEG-DASH standard, the encapsulated media files generated according to the traditional ISO Base Media File Format only allow players to achieve a single haptic interaction on a single screen, and cannot meet the needs of different haptic interactions in different areas of the screen.

[0027] Illustratively, the application of encapsulated media files in different haptic interaction scenes in different areas of the screen is illustrated in the following three scenes:

[0028] Scene 1: Game scene Taking a gunfight game as an example, when a character on the screen "fires a gun" in a certain tactile area, triggering a tactile interaction method corresponding to that tactile area (e.g., triggering a vibration motor on the right side of the mobile phone to vibrate) can greatly enhance the sense of immersion in the game. This triggering process can be viewed as the mobile phone receiving a command to "trigger the tactile actuator corresponding to that tactile area," then finding that tactile actuator and using that tactile actuator to "output" a vibration signal corresponding to that tactile area. Because games involve media file transmission processes such as downloading and updating, the vibration signal must be expressed in the encapsulated media file for each tactile area according to certain standards so that the video production device (i.e., game production device) and the player can transmit and analyze the encapsulated media file.

[0029] Scene 2: Movie scene For example, a company specializing in the manufacture of vibration chairs has a primary customer base of movie theaters offering immersive movie experiences. It has been discovered that providing haptic feedback in the field of movies (especially 4D movies) can bring new experiences to moviegoers and differentiate product capabilities to movie theaters. Ideally, such movies require high-quality video, audio, and haptic signal sources encapsulated within an encrypted carrier. To meet the need for synchronizing various signals (e.g., different haptic interactions in different areas of the screen), new encapsulation technologies are needed.

[0030] Furthermore, in the gaming field, especially in role-playing games (RPGs), story videos are often inserted during scene transitions to allow players to immerse themselves in the game's story and experience the character's behavior and emotional changes. When a game character in the inserted video stands up, opens a door, explodes in the surrounding scene, or runs, various haptic vibration feedback can be provided to enhance the story experience. Therefore, adding such haptic feedback to movie output on mobile devices is a very useful application scenario, but it also requires encapsulation of haptic signals (especially encapsulation of different haptic signals corresponding to different haptic regions). It can be understood that haptic signals themselves are similar to audio signals. Furthermore, the encapsulation of haptic signals can be thought of as a video file encapsulating the audio signal once and then encapsulating the haptic signal once to generate an encapsulated media file.

[0031] Scene 3: Remote surgery scene In the 5G era, remote surgery cases have been frequently reported, with the primary demand being the transmission of ultra-high-definition video at the surgical site. 5G networks can provide high-bandwidth, high-quality network services anytime, anywhere, making remote surgery applications a reality. However, a major difference between remote surgery and on-site surgery lies in tactile feedback. The ear is our primary auditory sensory organ, the eye is our primary visual sensory organ, and skin tissue is a tactile sensory organ that can receive various signals, such as heat and cold, smoothness and roughness, softness and hardness, pain and itch. Without these tactile feedback signals, remote surgeons would be unable to receive any tactile sensory signals, such as the softness and hardness of the patient's surgical lesion, tactile feedback when the operating equipment encounters an obstacle, or feedback on the smoothness and roughness of the opening and closing of surgical instruments. The lack of any of these sensory signals would affect the progress of the surgery. Therefore, tactile feedback must be added to the remote surgery signal during the transmission process, and different tactile signals corresponding to different tactile regions must be encapsulated.

[0032] In response to the problems in the related art, the present disclosure designs a method for encapsulating media files containing haptic signals, mainly solving the problems of encapsulating multiple haptic signal tracks, when haptic signal tracks are mounted, and the mapping between haptic areas and haptic signal tracks, enabling different haptic signals corresponding to different areas of a media file to be encapsulated, thereby achieving the effect of flexible analysis by players.

[0033] In the first aspect, referring to FIG. 2, there is shown a flowchart of a media file processing method according to an embodiment of the present disclosure, which may include the following steps.

[0034] Step S101: Determine a grayscale image track and each haptic signal track corresponding to the video data.

[0035] Here, the grayscale image track includes grayscale images corresponding to each of the frame screens in the video data, and the grayscale images corresponding to each of the frame screens represent the positional distribution of each of the tactile areas in each of the frame screens, and one tactile signal track corresponds to at least one grayscale value in the grayscale image, and the tactile signal track includes tactile signals corresponding to each of the tactile areas in the video data.

[0036] In a specific implementation, for each frame screen in the video data, the video production device divides a plurality of tactile regions from the frame screen and establishes tactile signal tracks corresponding to the plurality of tactile regions. After completing the establishment of the tactile signal tracks, the video production device may assign different grayscale values ​​to different tactile signal tracks, for example, assign one or more grayscale values ​​to a tactile signal track, where the one or more grayscale values ​​assigned to the tactile signal track describe the positional distribution in the video data of the tactile regions corresponding to the tactile signal track. Specifically, for any frame screen, at least one grayscale image is generated to represent the positional distribution of each tactile region in the frame screen based on the mapping relationship between the tactile signal tracks and the tactile regions in the frame screen and the grayscale values ​​assigned to each tactile signal track. After generating the grayscale images corresponding to each frame screen in the video data, the video production device may encapsulate the grayscale images corresponding to each frame screen in a grayscale image track according to the ISO base media file format.

[0037] Step S102: Generate a first mapping table based on each grayscale value in the grayscale image track and each haptic signal track.

[0038] In a specific implementation, the video production equipment calculates the grayscale values ​​contained in each grayscale image in the grayscale image track, determines the haptic signal track corresponding to each of the grayscale values, and generates a first mapping table representing the mapping relationship between each grayscale value and each haptic signal track.

[0039] Step S103: Encapsulate the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

[0040] Here, the grayscale image track can be thought of as a very colorless video, where each frame screen (i.e., each grayscale image) is very flat, with color jumps only at the edges between the grayscale image regions corresponding to the two haptic signal tracks. However, to minimize the transmission files required between the video production equipment and the player, the grayscale image track can be encoded, i.e., the grayscale image track, each haptic signal track, and the video data can be encoded and compressed before being encapsulated, thereby reducing the size of the target media file that needs to be transmitted.

[0041] It can be understood that after decapsulating a target media file to obtain a grayscale image track, the video output device can know the position distribution of each tactile area in each frame screen of the video data based on each grayscale image in the grayscale image track, but because the tactile signal tracks corresponding to different tactile areas are different, if the video output device does not know the mapping relationship between the tactile areas and the tactile signal tracks, it cannot correctly mount the tactile signal tracks to achieve different tactile interactions in different areas of the screen. Therefore, in the present disclosure, by adding a first mapping table to the target media file, the video output device can analyze the grayscale images in the grayscale image track and then determine the tactile signal tracks corresponding to each grayscale value in the grayscale image based on the first mapping table, and can correctly mount the tactile signal tracks for different tactile areas.

[0042] As can be seen from the above steps, the grayscale image track is encapsulated together with the haptic signal track and the video data into a target media file, thereby realizing the representation of the position distribution of each haptic area of ​​each frame screen in the video data in the target media file, and the first mapping table is added to the target media file, thereby realizing the representation of the mapping relationship between each grayscale value in the grayscale image track and each haptic signal track in the target media file, and further realizing the encapsulation of haptic signals by haptic area in the target media file, which can meet the different haptic interaction needs of different areas of the screen.

[0043] Embodiment 1 In this embodiment, an example of the first mapping table will be described.

[0044] Considering that after the player analyzes the grayscale image from the grayscale image track, the player needs to mount a haptic signal track corresponding to the analyzed grayscale image, in order to solve the problem of mapping between the analyzed grayscale image and the haptic signal track by the player, the present disclosure proposes a first mapping table, which is added to the target media file to inform the player of the haptic signal track corresponding to the analyzed grayscale image.

[0045] In this embodiment, as shown in FIG. 3, the player acquires and decapsulates the target media file to obtain video data, a first mapping table, grayscale image tracks corresponding to the video data, and haptic signal tracks. Then, the player outputs the video data and mounts the grayscale image tracks. Based on the grayscale image tracks and the first mapping table, the player mounts haptic signal tracks corresponding to the currently output grayscale image in the grayscale image tracks. The player then outputs haptic signals in the haptic signal tracks corresponding to the currently output grayscale image. Specifically, the player extracts a grayscale image (or a grayscale image sample, i.e., a grayscale sample) from the mounted grayscale image track, analyzes the grayscale values ​​of the grayscale image based on the first mapping table, and finds and mounts the corresponding haptic signal tracks. The grayscale image samples will be described in detail in embodiment 3, and will not be described in detail here.

[0046] In one possible embodiment, in response to the detection of a haptic interaction operation for a currently output screen in the video data, for example, when a touch component detects a haptic interaction operation such as a touch operation and sends a request to the player, the player finds a sample (i.e., a haptic signal sample) corresponding to the currently output screen among the currently mounted haptic signal tracks corresponding to the haptic area where the haptic interaction operation exists, and triggers the haptic actuator to output a haptic signal at the sample corresponding to the currently output screen. Here, the video production device is a device with encoding capabilities, and the player is a device with decoding capabilities. The video production device, player, touch component, and haptic actuator may be located in the same hardware device or may be located separately, and the present disclosure is not limited thereto.

[0047] In one embodiment, the target media file includes a box of a grayscale image track, and the video production equipment adds the first mapping table to the box of the grayscale track to achieve encapsulation of the first mapping table.

[0048] In one possible embodiment, the video production device generates a first target box based on a first mapping table, and adds the first target box to a box of the grayscale image track.

[0049] In this embodiment, the present disclosure newly proposes a Box type, i.e., a Box for storing the mapping relationship between grayscale values ​​and tracks (Gray_Value_To_Track Box, abbreviated as gvtt), to add a first mapping table to a target media file. The gvtt can connect the grayscale values ​​in a grayscale image track with other signal tracks (e.g., haptic signal tracks, audio tracks, etc.). By adding the gvtt (i.e., the first target box) storing the first mapping table to the box of the grayscale track, a player can directly obtain the first mapping table associated with the grayscale track when analyzing the grayscale track (i.e., analyzing the box of the grayscale track), and then correctly mount the haptic signal track based on the first mapping table.

[0050] In one possible embodiment, a first mapping table is generated based on the grayscale values ​​corresponding to each of the haptic signal tracks and the track index of each of the haptic signal tracks.

[0051] In a specific implementation, the basic structure of gvtt may be defined as a table consisting of grayscale values ​​and corresponding track indexes of other signal tracks. For example, the syntax of gvtt is defined as follows: aligned(8) class GrayValueToTrackBox extends FullBox('gvtt', version, flags){ unsigned int(32) gray_scale; / / Grayscale value in grayscale image track unsigned int(32) track_ID; / / Track index of other signal track }

[0052] After analyzing the grayscale value from the box of the grayscale image track, the player can determine the track index corresponding to the grayscale value based on the first mapping table stored in the first target box, and then find and mount the corresponding tactile signal track based on the track index.

[0053] Embodiment 2 In this embodiment, an example of the second mapping table will be described.

[0054] When outputting video, the player needs to mount and output a haptic signal track, and if there are multiple haptic areas in the video data, there is a problem of switching between multiple haptic signal tracks. When switching tracks, the player needs to find and mount the correct haptic signal track in a timely manner so that it can then extract samples from the mounted haptic signal track in a timely manner and output haptic signals.

[0055] To avoid delays in haptic signal output, the player needs to find and mount the correct haptic signal track in a timely manner when switching tracks. While it is possible to mount the haptic signal tracks in a timely manner by pre-mounting multiple haptic signal tracks (e.g., by mounting all haptic signal tracks), it will occupy a large cache space and cause a delay when finding and mounting the haptic signal track for the first time when switching tracks. Therefore, in this disclosure, to solve the problem of when the haptic signal track is mounted, a second mapping table is proposed and added to the target media file to inform the player of the required mounting time sequence for each haptic signal track.

[0056] In this embodiment, the video production equipment generates a second mapping table based on the execution order of each haptic signal track, and adds the second mapping table to the target media file, where the second mapping table is for describing the mapping relationship between the haptic signal tracks and the execution order, and the execution order is determined by the order of the mount times required for the tracks.

[0057] The player decapsulates the target media file to obtain a second mapping table, determines the next haptic signal track that needs to be mounted based on the currently mounted haptic signal track and the second mapping table, and pre-caches the haptic signal track before switching tracks, thereby enabling the player to quickly find and mount the haptic signal track corresponding to the grayscale image currently being output in the grayscale image track from the cache at the time of subsequent track switching.

[0058] For example, at time T1, track C1 of the haptic signal tracks is mounted, the player continuously extracts samples from track C1, and determines the next haptic signal track that needs to be mounted as track C2 based on the second mapping table, and pre-caches track C2 at time T2 (i.e., before track switching), and at time T3 when tracks are switched (e.g., the time when the player receives a request to mount track C2), the player immediately analyzes the samples in track C2 in the cache to complete the mounting. As can be understood, the player can know which haptic signal tracks can be called at the current time based on the second mapping table, and can find and cache the haptic signal tracks that can be called in advance, which corresponds to realizing pre-mounting of the haptic signal track that is to be mounted.

[0059] In one embodiment, the video production device generates a second mapping table corresponding to each haptic signal track based on the sample index (this sample index may be a sample number, a sample address within a box, etc.) and execution order of the key frame of each haptic signal track. When the player analyzes the key frame in the currently mounted haptic signal track, it can determine the sample index of the key frame of the next haptic signal track that needs to be mounted based on the second mapping table. This allows the player to accurately know the chunk that can be called (a haptic signal track is made up of at least one chunk) based on the sample index, and further determine the sample position in the chunk that can be called that corresponds to the current time, and perform mounting.

[0060] As one possible embodiment, as shown in Fig. 4, a key frame of a haptic signal track is determined as the starting sample of the haptic signal track, and a second mapping table is generated based on the sample index and execution order of the starting sample of each haptic signal track. Illustratively, the syntax of the second mapping table is defined as follows: aligned(8) class SyncHapticBox extends FullBox('shss', version = 0, 0) { unsigned int(32) order_count; / / Execution order of the starting sample int i; for (i=0; i < order_count; i++) { unsigned int(32) sample_number; / / Sample index of the starting sample } }

[0061] In one embodiment, if the target media file includes a box for each haptic signal track, a second mapping table corresponding to each haptic signal track can be added to each box, where the second mapping table corresponding to a haptic signal track refers to a second mapping table that describes at least the execution order of that haptic signal track and the next haptic signal track that needs to be mounted.

[0062] In this disclosure, a Box type, i.e., a Box for storing a mapping relationship between sample indexes and sample execution orders (Haptic Time To Sample Box, abbreviated as https), is newly proposed to add a first mapping table to a box of a corresponding haptic signal track. The video production equipment may generate https (i.e., second target boxes) corresponding to each haptic signal track based on a second mapping table corresponding to each haptic signal track, and add each second target box to a box of the corresponding haptic signal track.

[0063] Embodiment 3 In this embodiment, an example of a grayscale image in a grayscale image track will be described.

[0064] The grayscale image track can encapsulate grayscale images corresponding to each frame screen in the video data in the form of grayscale image samples (i.e., grayscale samples), where one grayscale image track is composed of multiple grayscale image samples including at least one grayscale image, whereby the player can extract the grayscale image sample corresponding to at least one frame screen in the video data (e.g., a portion of the video) from the mounted grayscale image track.

[0065] Case 1: One grayscale image represents the position distribution of all or part of the tactile areas corresponding to one frame screen in the video data.

[0066] 5, for one frame screen including three tactile regions, the grayscale image corresponding to the frame screen may include grayscale regions corresponding to three different grayscale values, and the positional distribution of the three grayscale regions in the grayscale image is the same as the positional distribution of the corresponding three tactile regions in the frame screen.

[0067] An application scene of the above case 1 will be explained using a movie scene as an example.

[0068] For a 4D movie designed for haptic interaction, when encapsulating a media file for the 4D movie, one or more grayscale images corresponding to each frame screen are generated based on the tactile regions included in each frame screen of the video data of the 4D movie. For example, for a frame screen including three tactile regions, one grayscale image representing the positional distribution of the three tactile regions in the frame screen may be generated. Alternatively, two grayscale images may be generated: one grayscale image representing the positional distribution of one tactile region in the frame screen and the other grayscale image representing the positional distribution of the remaining two tactile regions in the frame screen. The one or two generated grayscale images are encapsulated as grayscale image samples corresponding to the frame screen, and a grayscale image track corresponding to the video data is generated based on the grayscale image samples corresponding to each frame screen of the video data.

[0069] When generating a grayscale image corresponding to a frame screen, it is necessary to assign at least one grayscale value to each tactile region in the frame screen, and then generate the corresponding grayscale image based on the grayscale values ​​assigned to the tactile region. For example, if the grayscale values ​​assigned to a certain tactile region are 0 and 1, first determine the positional distribution of the tactile region in the frame screen, and set the grayscale value of each pixel point in the same positional distribution in the grayscale image to 0 or 1. The positional distribution of the tactile region in the frame screen can be represented by the positional distribution in the grayscale image of the grayscale region composed of the grayscale values ​​of 0 and 1. After the tactile regions are assigned grayscale values, a mapping relationship between the grayscale values ​​assigned to each tactile region and the tactile signal track corresponding to that tactile region can be recorded in a first mapping table.

[0070] The first mapping table, the grayscale track, and the video data are encapsulated as a target media file and transmitted to a player in a cinema. The player then decapsulates the target media file, outputs the video data, and mounts the grayscale image track. The player can then present the output video data through a projection device in the cinema. The mounted grayscale image track outputs grayscale images in chronological order. The player then finds and mounts a corresponding haptic signal track according to the grayscale image currently output by the projection device (i.e., the grayscale image currently output in the grayscale image track) using the first mapping table. The mounted haptic signal track outputs haptic signals in chronological order. The player then uses a haptic actuator (e.g., a vibrating chair) in the cinema to present the output haptic signals.

[0071] As can be understood, depending on the screen fluctuations that occur during the output cycle of the video image (i.e., fluctuations in the number and positional distribution of the presented tactile areas), the number and positional distribution of the grayscale areas presented by the grayscale track must also fluctuate accordingly.

[0072] Case 2: One grayscale image represents the position distribution of one tactile area corresponding to one frame screen in the video data.

[0073] The video player determines each haptic signal source corresponding to the video data, with one haptic signal source corresponding to one haptic signal track. A haptic area corresponding to each haptic signal source is determined based on a corresponding area of ​​the haptic signal of each haptic signal source in each frame screen of the video data, where the haptic area may be the entire screen area. A grayscale image track corresponding to each haptic signal source is generated based on the grayscale value and haptic area corresponding to each haptic signal source, with one grayscale image track representing the positional distribution in the video data of the haptic area corresponding to one haptic signal source.

[0074] We will explain the application scenario of Case 2 above using a remote surgery scenario as an example. In the case of a remote surgery scenario, the needs can be categorized as follows: Need 1: Multiple tactile signal tracks Need 2: Unlike pre-created scenes, which compress and then output videos containing tactile information, remote surgery primarily involves presenting tactile signals in real-time scenes. This makes it impossible to achieve track matching by pre-creating grayscale images that can represent the location distribution of multiple tactile areas for each tactile signal track. Needs 3: Not only tactile information from vibration, but also feedback information on ergonomic forces is included.

[0075] To address the above needs, as shown in FIG. 6 , the present disclosure uses a scalpel and surgical scissors as two tactile signal sources, and constructs two grayscale image tracks and two tactile signal tracks. One tactile signal source corresponds to one grayscale image track and one tactile signal track, and one grayscale image track may contain only one grayscale value. Taking the scalpel as an example, the grayscale image presented in the grayscale image track corresponding to the scalpel is used to represent the position of the scalpel (i.e., the tactile area corresponding to the scalpel) on the currently output video screen, and the tactile signal presented in the tactile signal track corresponding to the scalpel is the tactile signal of the corresponding tactile area. The tactile signal is generated based on ergonomic force feedback information of the scalpel position on the currently output video screen. As the position of the scalpel changes on the video screen, the tactile signal track of the scalpel describes ergonomic force feedback information in real time, such as feedback on the softness and hardness of the surgical lesion tissue at the scalpel position and tactile feedback that the scalpel is obstructed.

[0076] Before surgery, the player tests whether the connection relationships established between the scalpel and surgical scissors and the grayscale track and tactile signal track are correct, and can correct them in time to avoid incorrect connection situations, such as connecting the grayscale track corresponding to the surgical scissors with the scalpel. In the target encapsulation file transmitted between the encapsulation side (i.e., the video production side where the patient is) and the decapsulation side (i.e., the video playback side where the doctor is), the grayscale image is very flat and the tactile signals are represented by two grayscale tracks with a single color, so the bandwidth required to transmit the target encapsulation file is relatively small, and the low-latency needs of remote surgery scenes can be well met.

[0077] Embodiment 4 In this embodiment, an example of allocation of haptic actuators will be described.

[0078] After the player loads the target media file, it first needs to know how many haptic tracks the file contains in total, so as to allocate relevant resources reasonably in advance (e.g., allocate the player's haptic actuators). Therefore, the present disclosure designs to add the number of haptic areas in the video data or the number of grayscale values ​​in the grayscale image track to the target media file.

[0079] Case 1: The video production device counts the number of tactile regions in the video data and adds the number of tactile regions to the target media file.

[0080] In this embodiment, after encapsulating the target media file, the player can determine the number of haptic signal tracks according to the number of haptic areas obtained, and thereby pre-allocate haptic actuators to each haptic signal track according to the number of haptic signal tracks, thereby realizing timely and reasonable allocation of haptic actuators, so that the player can then timely trigger the haptic actuators corresponding to the haptic signal tracks to output corresponding haptic signals.

[0081] Case 2: When one haptic signal track corresponds to one grayscale value in the grayscale image, the video production equipment counts the number of grayscale values ​​in the grayscale image track and adds the number of grayscale values ​​to the target media file.

[0082] In this embodiment, after decapsulating the target media file, the player can determine the number of haptic signal tracks according to the number of grayscale values ​​obtained, and then assign a haptic actuator to each of the haptic signal tracks according to the number of haptic signal tracks, and select the haptic actuator corresponding to the haptic signal track to output a haptic signal. This avoids the player not knowing the number of haptic signal tracks and assigning all the haptic signal tracks of the target media file to the same haptic actuator, which would result in a poor presentation effect of the haptic signal.

[0083] In one possible embodiment, if the target media file includes a box for a grayscale image track, the number of grayscale values ​​for the grayscale image track may be added to the box for the grayscale image track, such as the media header or track header of the box for the grayscale image track.

[0084] As can be understood, in accordance with the classification method of track-chunk-Sample, in order to encapsulate grayscale value track description information (e.g., the number of grayscale values ​​in the grayscale track) in the grayscale track, content can be added to the track header (tkhdh, track header) of the grayscale track box or the media header (mdhd, media header) of the grayscale track box.

[0085] The media header is used to store overall information about the media, and a conventional media header contains the information in Table 1 below.

[0086] [Table 1]

[0087] The track header is used to store descriptive information about the current track, and the information contained in a conventional track header is shown in Table 2 below.

[0088] [Table 2-1] [Table 2-2] [Table 2-3]

[0089] In a second aspect, as shown in FIG. 7, an embodiment of the present disclosure provides another media file processing method, which includes at least the following steps. Step S201: Obtain and decapsulate a target media file to obtain video data, a first mapping table, a grayscale image track corresponding to the video data, and each haptic signal track. Here, the grayscale image track includes grayscale images corresponding to each frame image in the video data, and the grayscale images corresponding to each frame image represent the positional distribution of the tactile area of ​​each frame image, and one tactile signal track corresponds to at least one grayscale value in the grayscale images. Step S202: Output the video data and mount the grayscale image track. Step S203: Mount a haptic signal track corresponding to the grayscale image currently output in the grayscale image track according to the grayscale image track and the first mapping table. Step S204: Output a haptic signal in a haptic signal track corresponding to the currently output grayscale image.

[0090] As can be seen from the above steps, the grayscale image track is encapsulated together with the haptic signal track and the video data into a target media file, thereby realizing the representation of the position distribution of each haptic area of ​​each frame screen in the video data in the target media file, and the first mapping table is added to the target media file, thereby realizing the representation of the mapping relationship between each grayscale value in the grayscale image track and each haptic signal track in the target media file, and further realizing the encapsulation of haptic signals by haptic area in the target media file, which can meet the different haptic interaction needs of different areas of the screen.

[0091] In one possible embodiment, when one haptic signal track corresponds to one grayscale value of a grayscale image, the decapsulated information of the target media file further includes the number of grayscale values ​​in the grayscale image track, and the method includes: determining the number of haptic signal tracks in response to the number of grayscale values; and assigning a haptic actuator to each of the haptic signal tracks in accordance with the number of the haptic signal tracks; The step of outputting a haptic signal in a sample corresponding to the currently output screen includes: Triggering a haptic actuator corresponding to a haptic signal track in which the sample is located to output a haptic signal at the sample.

[0092] In one possible embodiment, the decapsulated information of the target media file further includes a second mapping table describing a mapping relationship between a haptic signal track and an execution order, and the method includes: determining a next haptic signal track that needs to be mounted based on the currently mounted haptic signal track and the second mapping table, and caching the haptic signal track; Mounting a haptic signal track corresponding to each grayscale image in the grayscale image track based on the grayscale image track and the first mapping table includes: The method includes mounting, from a cache, a haptic signal track corresponding to the grayscale image currently being output in the grayscale image track based on the grayscale image track and the first mapping table.

[0093] The media file processing method according to the embodiment of the present disclosure may be executed by a media file processing device. In the embodiment of the present disclosure, the media file processing device according to the embodiment of the present disclosure will be described using an example in which the media file processing device executes the media file processing method.

[0094] In a third aspect, an embodiment of the present disclosure provides a media file processing device, as shown in FIG. 8, the media file processing device 100 includes: a first determination module 101 for determining a grayscale image track and each haptic signal track corresponding to video data, wherein the grayscale image track includes a grayscale image corresponding to each frame screen in the video data, the grayscale image corresponding to each frame screen represents a position distribution of each tactile area in each frame screen, and the haptic signal track includes a haptic signal corresponding to each tactile area in the video data; a first generating module 102 that generates a first mapping table based on each grayscale value in the grayscale image track and each haptic signal track; a first encapsulation module 103 for encapsulating the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

[0095] Alternatively, the first generating module 102 may: A first generating sub-module is included that generates a first mapping table based on the grayscale values ​​corresponding to each of the haptic signal tracks and the track indexes of each of the haptic signal tracks.

[0096] Alternatively, the target media file includes a box of grayscale image tracks, and the first encapsulation module 103: A second encapsulation module is included that adds the first mapping table to a box of the grayscale image track.

[0097] Alternatively, the second encapsulation module may include: a first encapsulation sub-module that generates a first target box based on the first mapping table; a second encapsulation sub-module that adds the first target box to a box of the grayscale image track.

[0098] Alternatively, the device comprises: a first statistics module for calculating a number of tactile regions in the video data; The first encapsulation module 103 and a third encapsulation module for encapsulating the number of haptic areas, the first mapping table, the grayscale image track, the respective haptic signal tracks, and the video data to obtain a target media file.

[0099] Alternatively, if one haptic signal track corresponds to one grayscale value in the grayscale image, the device: a second statistics module for calculating a number of grayscale values ​​in the grayscale image track; The first encapsulation module 103 and a fourth encapsulation module for encapsulating the number of grayscale values, the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

[0100] Alternatively, the target media file includes a box of grayscale image tracks, and the fourth encapsulation module: and a third encapsulation sub-module for adding the number of grayscale values ​​to a box in the grayscale image track.

[0101] Alternatively, the third encapsulation sub-module: and a fourth encapsulation sub-module that adds the number of grayscale values ​​to a media header or a track header of a box of the grayscale image track.

[0102] Alternatively, the device comprises: a second generating module configured to generate a second mapping table based on an execution order of each of the haptic signal tracks; The first encapsulation module 103 a fifth encapsulation module for encapsulating the second mapping table, the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file;

[0103] Alternatively, the second generation module and a second generating sub-module that generates a second mapping table corresponding to each of the haptic signal tracks based on the sample indexes and execution orders of the key frames of each of the haptic signal tracks.

[0104] Alternatively, the keyframe of the haptic signal track is the starting sample of the haptic signal track.

[0105] Alternatively, the target media file includes a box for each of the haptic signal tracks, and the fifth encapsulation module: and a fifth encapsulation sub-module that adds a second mapping table corresponding to each of the haptic signal tracks to each box.

[0106] Alternatively, the fifth encapsulation sub-module may include: a sixth encapsulation sub-module that generates second target boxes corresponding to each of the haptic signal tracks based on second mapping tables corresponding to each of the haptic signal tracks; and a seventh encapsulation sub-module that adds each of the second target boxes to a box of the corresponding haptic signal track.

[0107] Alternatively, the first determination module 101 a first determining sub-module for determining each haptic signal source corresponding to the video data, where one haptic signal source corresponds to one haptic signal track; a second determination sub-module for determining a tactile area corresponding to each of the tactile signal sources based on a corresponding area of ​​a tactile signal of each of the tactile signal sources in each frame screen of the video data; and a third determination sub-module that generates a grayscale image track corresponding to each of the haptic signal sources based on the grayscale value and the haptic area corresponding to each of the haptic signal sources.

[0108] The media file processing device according to the embodiment of the present disclosure can implement each process implemented in the embodiment of the media file processing method described in the first aspect and can achieve the same technical effects, so to avoid duplication, they will not be described in detail here.

[0109] In a fourth aspect, an embodiment of the present disclosure provides another media file processing device, as shown in FIG. 9, the media file processing device 200 includes: a first processing module 201 for acquiring and decapsulating a target media file to obtain video data, a first mapping table, a grayscale image track corresponding to the video data, and each haptic signal track, wherein the grayscale image track includes a grayscale image corresponding to each frame screen in the video data, and the grayscale image corresponding to each frame screen represents a position distribution of each haptic area in each frame screen; a second processing module 202 for outputting the video data and mounting the grayscale image track; a third processing module 203 for mounting a haptic signal track corresponding to the grayscale image currently output in the grayscale image track according to the grayscale image track and the first mapping table; and a fourth processing module 204 that outputs a haptic signal in a haptic signal track that corresponds to the currently output grayscale image.

[0110] Alternatively, if one haptic signal track corresponds to one grayscale value in the grayscale image, the information decapsulating the target media file further includes the number of grayscale values ​​in the grayscale image track, and the device: a fifth processing module that determines the number of haptic signal tracks in response to the number of grayscale values; a sixth processing module that assigns a haptic actuator to each of the haptic signal tracks in accordance with the number of the haptic signal tracks; The fourth processing module 204 A first processing sub-module is included that triggers a haptic actuator corresponding to a haptic signal track on which the sample is located to output a haptic signal at the sample.

[0111] Alternatively, the decapsulated information of the target media file further includes a second mapping table describing a mapping relationship between a haptic signal track and an execution order, and the device: a seventh processing module that determines a next haptic signal track that needs to be mounted based on a currently mounted haptic signal track and the second mapping table, and caches the haptic signal track; The third processing module 203 and a second processing sub-module that mounts from a cache a haptic signal track corresponding to the grayscale image currently being output in the grayscale image track based on the grayscale image track and the first mapping table.

[0112] Alternatively, the fourth processing module 204 may and a third processing sub-module that, in response to a haptic interaction operation being detected for a currently output screen in the video data, outputs a haptic signal corresponding to the currently output screen from a haptic signal track corresponding to a haptic area in which the currently mounted haptic interaction operation exists. The media file processing device according to the embodiment of the present disclosure can implement each process implemented in the embodiment of the media file processing method described in the second aspect and can achieve the same technical effects, so to avoid duplication, they will not be described in detail here.

[0113] The above-described device embodiments are merely schematic, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network elements. Depending on actual needs, some or all of these modules may be selected to achieve the objectives of the aspects of the present embodiment. Those skilled in the art can understand and implement the present embodiment without any creative effort.

[0114] Each component of the present disclosure may be implemented in hardware, software modules running on one or more processors, or a combination thereof. Those skilled in the art should understand that a microprocessor or digital signal processor (DSP) may be practically used to implement some or all of the functions of some or all of the components of a computing device according to an embodiment of the present disclosure. The present disclosure may also be implemented as an apparatus or device program (e.g., a computer program and computer program product) for executing some or all of the methods described herein. Such a program implementing the present disclosure may be stored on a computer-readable medium or may have the form of one or more signals. Such signals may be downloaded from an Internet website, provided on a carrier signal, or provided in any other format.

[0115] For example, FIG. 10 illustrates a computing device capable of implementing the methods of the present disclosure. The computing device conventionally includes a processor 1010 and a computer program product or computer-readable medium in the form of memory 1020. Memory 1020 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Memory 1020 has a storage space 1030 for program code 1031 for performing the method steps of any of the above methods. For example, program code storage space 1030 may include program code 1031 used to implement various steps of the above methods, respectively. These program codes may be read from or written to one or more computer program products. These computer program products include program code carriers such as a hard disk, a compact disc (CD), a memory card, or a floppy disk. Such computer program products are typically portable or fixed storage units, as described with reference to FIG. 11. The storage unit may have storage segments, storage spaces, etc. arranged similarly to memory 1020 of the computing device of Figure 10. The program code may, for example, be compressed in a suitable format. Typically, the storage unit contains computer-readable code 1031', i.e., code that can be read by a processor, such as 1010, which, when executed by the computing device, causes the computing device to perform the steps of the methods described above.

[0116] An embodiment of the present disclosure further provides a media file processing device that includes a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, realizing each process of the above-mentioned embodiments of the media file processing method, and can achieve the same technical effects, so to avoid duplication, they will not be described in detail here.

[0117] The embodiments of the present disclosure further provide a readable storage medium storing a program or instruction that, when executed by a processor, realizes each process of the embodiments of the above media file processing method, and can achieve the same technical effect, so to avoid duplication, they will not be described in detail here.

[0118] The processor is the processor of the terminal device described in the above embodiments. The readable storage medium includes a computer readable storage medium such as a computer read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] An embodiment of the present disclosure includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor further provides a chip that executes a program or instruction to realize each process of the above-mentioned embodiment of the media file processing method, and can achieve the same technical effect, so to avoid duplication, it will not be described in detail here.

[0120] The chips referred to in the embodiments of the present disclosure are also called system level chips, system chips, chip systems, or system-on-chips.

[0121] The embodiments of the present disclosure further provide a computer program / program product that is stored in a storage medium and executed by at least one processor to realize each process of the above-mentioned embodiments of the media file processing method, and can achieve the same technical effects, so to avoid duplication, it will not be described in detail here.

[0122] An embodiment of the present disclosure further provides a media file processing system including a video production device for performing the steps of the media file processing method described in the first aspect above, and a video output device for performing the steps of the media file processing method described in the second aspect above.

[0123] It should be noted that, as used herein, the terms "comprise," "include," and any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus comprising a set of elements includes not only those elements but also other elements not expressly described or elements inherent in such a process, method, article, or apparatus. Unless further limited, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, the scope of the methods and apparatuses in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, but may include performing functions substantially simultaneously or in reverse order, depending on the functionality involved, for example, in an order different from that described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to particular examples may be combined in other examples.

[0124] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be realized through software and a necessary general-purpose hardware platform, and of course through hardware, and in many cases, the former is a more preferred embodiment. Based on this understanding, the essential or relevant parts of the technical solutions of the present disclosure may be embodied in the form of a computer software product, which may be stored in a storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and include multiple instructions that cause a terminal (which may be a mobile phone, computer, server, air conditioner, network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0125] Although the examples of the present disclosure have been described above with reference to the drawings, the present disclosure is not limited to the above-mentioned specific embodiments, which are schematic and not restrictive. Those skilled in the art can create many more forms under the inspiration of the present disclosure without departing from the spirit of the present disclosure and the scope of protection of the claims, and all of these fall within the scope of protection of the present disclosure.

[0126] As used herein, the terms "one embodiment," "an embodiment," or "one or more embodiments" mean that a particular feature, structure, or characteristic described with reference to an embodiment is included in at least one embodiment of the present disclosure. Note that instances of the phrase "in one embodiment" do not necessarily refer to the same embodiment.

[0127] In the description provided herein, many specific details are set forth. However, it will be understood that embodiments of the present disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail in order not to obscure an understanding of this specification.

[0128] In the claims, any reference signs placed between parentheses do not limit the scope of the claim. The word "comprises" does not exclude the presence of elements or steps not stated in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present disclosure may be implemented by hardware comprising several different elements, and by a suitably programmed computer. In a unit claim enumerating several devices, several of these devices may be embodied by the same item of hardware. The use of terms such as first, second, and third does not indicate any order. These words may be interpreted as names.

[0129] Finally, the above embodiments are used to explain the technical solutions of the present disclosure, but are not intended to limit the same. Although the present disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may be modified or some technical features may be replaced with equivalents. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure. [Explanation of symbols]

[0130] 100 Media file processing device 101 First Decision Module 102 First Generation Module 103 First Encapsulation Module 200 Media file processing device 201 First Processing Module 202 Second Processing Module 203 Third Processing Module 204 Fourth Processing Module 1010 processor 1020 memory 1030 Storage Space 1031 Program Code 1031' computer readable code

Claims

1. 1. A method for processing a media file, comprising: determining a grayscale image track and each haptic signal track corresponding to video data, wherein the grayscale image track includes a grayscale image corresponding to each frame screen in the video data, the grayscale image corresponding to each frame screen representing a positional distribution of each tactile area in each frame screen, and the haptic signal track includes a haptic signal corresponding to each tactile area in the video data; generating a first mapping table based on each grayscale value in the grayscale image track and each haptic signal track; encapsulating the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

2. generating a first mapping table based on each grayscale value in the grayscale image track and each haptic signal track, The method of claim 1 , further comprising generating a first mapping table based on a grayscale value corresponding to each of the haptic signal tracks and a track index of each of the haptic signal tracks.

3. the target media file includes a box of grayscale image tracks; The first mapping table comprises: The method of claim 1 , wherein the first mapping table is encapsulated in the target media file by adding the first mapping table to a box in the grayscale image track.

4. The step of adding the first mapping table to a box of the grayscale image track includes: generating a first target box based on the first mapping table; and adding the first target box to a box of the grayscale image track.

5. further comprising the step of calculating a number of tactile regions in the video data; encapsulating the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file, the step of: The method of claim 1 , comprising encapsulating the number of haptic areas, the first mapping table, the grayscale image track, the respective haptic signal tracks, and the video data to obtain a target media file.

6. If one haptic signal track corresponds to one grayscale value in the grayscale image, the method comprises: and further comprising the step of calculating a number of grayscale values ​​in the grayscale image track. encapsulating the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file, the step of:

2. The method of claim 1, comprising encapsulating the number of grayscale values, the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

7. the target media file includes a box of grayscale image tracks; The number of grayscale values ​​is The method of claim 6 , wherein the number of grayscale values ​​is encapsulated in the target media file by adding the number of grayscale values ​​to a box in the grayscale image track.

8. adding the number of grayscale values ​​to a box in the grayscale image track; The method of claim 7 , further comprising adding the number of grayscale values ​​to a media header or track header of a box of the grayscale image track.

9. generating a second mapping table based on the mapping relationship between the haptic signal track and the execution order; encapsulating the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file, the step of:

2. The method of claim 1, further comprising encapsulating the second mapping table, the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

10. generating a second mapping table based on a mapping relationship between the haptic signal track and the execution order, 10. The method of claim 9, further comprising generating a second mapping table corresponding to each of the haptic signal tracks based on a mapping relationship between sample indexes and execution orders of key frames of each of the haptic signal tracks.

11. The method of claim 10 , wherein the keyframe of the haptic signal track is the starting sample of the haptic signal track.

12. the target media file includes a box for each of the haptic signal tracks; The second mapping table is The method of claim 10 , wherein the second mapping table corresponding to each of the haptic signal tracks is encapsulated in the target media file by adding a respective box.

13. adding a second mapping table corresponding to each of the haptic signal tracks to each box, generating second target boxes corresponding to each of the haptic signal tracks based on a second mapping table corresponding to each of the haptic signal tracks; and adding each said second target box to a box of a corresponding haptic signal track.

14. determining a grayscale image track corresponding to the video data; determining each haptic signal source corresponding to the video data, one haptic signal source corresponding to one haptic signal track; determining a tactile area corresponding to each of the tactile signal sources based on a corresponding area of ​​the tactile signal of each of the tactile signal sources in each frame screen of the video data; and generating a grayscale image track corresponding to each of the haptic signal sources based on the grayscale value and the tactile area corresponding to each of the haptic signal sources.

15. 1. A method for processing a media file, comprising: acquiring and decapsulating the target media file to obtain video data, a first mapping table, a grayscale image track corresponding to the video data, and each haptic signal track, wherein the grayscale image track includes a grayscale image corresponding to each frame screen in the video data, and the grayscale image corresponding to each frame screen represents a position distribution of each haptic area in each frame screen; outputting the video data and mounting the grayscale image track; Mounting a haptic signal track corresponding to the grayscale image currently being output in the grayscale image track based on the grayscale image track and the first mapping table; outputting a haptic signal in a haptic signal track corresponding to the currently output grayscale image.

16. The decapsulated information of the target media file further includes a number of tactile regions in the video data, and the method further comprises: determining the number of haptic signal tracks in response to the number of haptic areas; and assigning a haptic actuator to each of the haptic signal tracks in accordance with the number of the haptic signal tracks; outputting a haptic signal in a haptic signal track corresponding to the currently output grayscale image, The method of claim 15 , comprising triggering a haptic actuator corresponding to the haptic signal track to output the haptic signal.

17. When one haptic signal track corresponds to one grayscale value in the grayscale image, the decapsulated information of the target media file further includes the number of grayscale values ​​in the grayscale image track, and the method includes: determining the number of haptic signal tracks in response to the number of grayscale values; and assigning a haptic actuator to each of the haptic signal tracks in accordance with the number of the haptic signal tracks; outputting a haptic signal in a haptic signal track corresponding to the currently output grayscale image, The method of claim 15 , comprising triggering a haptic actuator corresponding to the haptic signal track to output the haptic signal.

18. The decapsulated information of the target media file further includes a second mapping table describing a mapping relationship between a haptic signal track and an execution order, and the method further comprises: determining a next haptic signal track that needs to be mounted based on the currently mounted haptic signal track and the second mapping table, and caching the haptic signal track; Mounting a haptic signal track corresponding to each grayscale image in the grayscale image track based on the grayscale image track and the first mapping table includes:

16. The method of claim 15, further comprising mounting from a cache a haptic signal track corresponding to the grayscale image currently being output in the grayscale image track based on the grayscale image track and the first mapping table.

19. outputting a haptic signal in a haptic signal track corresponding to the currently output grayscale image, A method according to any one of claims 15 to 18, comprising, in response to detection of a haptic interaction operation for a currently output screen in the video data, outputting a haptic signal corresponding to the currently output screen from a haptic signal track corresponding to a haptic area in which the currently mounted haptic interaction operation exists.

20. a first determination module that determines a grayscale image track and each haptic signal track corresponding to video data, the grayscale image track including grayscale images corresponding to each frame screen in the video data, the grayscale images corresponding to each frame screen representing a positional distribution of each tactile area in each frame screen, and the haptic signal tracks including haptic signals corresponding to each tactile area in the video data; a first generation module that generates a first mapping table based on each grayscale value in the grayscale image track and each haptic signal track; a first encapsulation module that encapsulates the first mapping table, the grayscale image track, the haptic signal tracks, and the video data to obtain a target media file.

21. a first processing module that acquires and decapsulates a target media file to obtain video data, a first mapping table, a grayscale image track corresponding to the video data, and each haptic signal track, wherein the grayscale image track includes a grayscale image corresponding to each frame screen in the video data, and the grayscale image corresponding to each frame screen represents a position distribution of each haptic area in each frame screen; a second processing module that outputs the video data and mounts the grayscale image track; a third processing module that mounts a haptic signal track corresponding to the grayscale image currently being output in the grayscale image track based on the grayscale image track and the first mapping table; a fourth processing module that outputs a haptic signal in a haptic signal track that corresponds to the currently output grayscale image.

22. A media file processing device comprising a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, realizing the steps of the media file processing method of any one of claims 1 to 14.

23. A media file processing device comprising a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, realizing the steps of the media file processing method of any one of claims 15 to 19.

24. a memory having computer readable code stored therein; one or more processors; A computing device that, when the computer-readable code is executed by the one or more processors, performs the media file processing method of any one of claims 1 to 14 or performs the media file processing method of any one of claims 15 to 19.

25. A computer program comprising computer readable code which, when executed on a computing device, causes the computing device to perform the media file processing method of any one of claims 1 to 14 or the media file processing method of any one of claims 15 to 19.

26. 26. A computer readable medium storing the computer program of claim 25.

Images