Multimedia file playing and generating method

Abstract

The invention discloses a method for playing and generating a multimedia file, and belongs to the technical field of multimedia. In the method, in response to an opening operation of the multimedia file, an image in the multimedia file is displayed, a sampling point sequence of steganographic audio is extracted from pixel information of the image, operation information is determined in response to a user operation for the image, and the operation information is sent to the multimedia file. And determining a sampling point corresponding to the operation information from the sampling point sequence according to the audio synchronization information in the multimedia file, and playing the audio according to the determined sampling point. Therefore, the multimedia file formed by combining the images and the audios contains the audio synchronization information, the synchronous playing mode of the images and the audios can be refined to the operation information of the images and the sampling points of the audios by means of the audio synchronization information, the refined synchronous playing mode is beneficial to improving the playing flexibility of the multimedia file, and the playing flexibility of the multimedia file is improved. The sampled audio is steganographically written in the pixel information of the image, and the size of the image is not changed, so that the size of the multimedia file is relatively small.

Description

Technical Field

[0001] The present application relates to the field of multimedia technology, and in particular to a method for playing and generating multimedia files. Background Art

[0002] In the prior art, when it is necessary to combine images and audio together, a tool is used to set the image as an animation frame, embed the audio file, and then output it to a video format. For example, using Adobe Premiere Pro or iMovie tools, the image is imported into the timeline, the duration is set, the audio file is added, and it is exported to a video. Subsequently, the image and audio are played in the form of a video. The problem is that the synchronous playback method of the image and audio in the video is single, and the video file is also relatively large. Summary of the invention

[0003] The embodiment of the present application provides a method for playing and generating a multimedia file, so as to solve the problem in the related art that when combining images and audio, the playback mode of the combined multimedia file is single and the multimedia file is relatively large.

[0004] In a first aspect, an embodiment of the present application provides a method for playing a multimedia file, comprising:

[0005] In response to an opening operation of a multimedia file, an image in the multimedia file is displayed, and a sampling point sequence of the steganographic audio is extracted from pixel information of the image;

[0006] determining operation information in response to a user operation on the image;

[0007] Determining, from the sampling point sequence, a sampling point corresponding to the operation information according to the audio synchronization information in the multimedia file, wherein the audio synchronization information is used to indicate a corresponding relationship between the operation information and the sampling points in the sampling point sequence;

[0008] The audio is played according to the determined sampling points.

[0009] In some embodiments, the pixel information of the image includes pixel values ​​of each pixel on multiple channels, and extracting a sampling point sequence of the steganographic audio from the pixel information of the image includes:

[0010] Extracting tail values ​​of pixel values ​​of at least some pixels on multiple channels;

[0011] The extracted values ​​are combined to obtain the sampling point sequence.

[0012] In some embodiments, the audio synchronization information is used to indicate a correspondence between an image region in the image and sampling points in the sampling point sequence, and the operation information includes a target image region in the image to be operated;

[0013] Determining the sampling point corresponding to the operation information from the sampling point sequence according to the audio synchronization information in the multimedia file includes:

[0014] According to the audio synchronization information, the sampling point corresponding to the target image area in the sampling point sequence is determined as the sampling point corresponding to the operation information.

[0015] In some embodiments, the audio synchronization information is used to indicate the corresponding relationship between the operation sequence of the plurality of image regions in the image and the sampling points in the sampling point sequence; the operation information includes the operation sequence of the plurality of target image regions in the image;

[0016] Determining the sampling point corresponding to the operation information from the sampling point sequence according to the audio synchronization information in the multimedia file includes:

[0017] According to the audio synchronization information, sampling points corresponding to the operation order of the plurality of target image regions in the sampling point sequence are determined as sampling points corresponding to the operation information.

[0018] In a second aspect, an embodiment of the present application provides a method for generating a multimedia file, comprising:

[0019] Acquire an image, audio, and synchronization description information, wherein the synchronization description information is used to describe the synchronization relationship between the operation information of the image and the audio content in the audio;

[0020] Sampling the audio to obtain a sampling point sequence, and steganographically writing the sampling points in the sampling point sequence into the pixel information of the image;

[0021] According to the synchronization description information, a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence is established to obtain audio synchronization information;

[0022] Based on the pixel information of the steganographic image and the audio synchronization information, a multimedia file is generated in a multimedia file format.

[0023] In some embodiments, the pixel information of the image includes pixel values ​​of each pixel on multiple channels, and the sampling points in the sampling point sequence are hidden in the pixel information of the image, including:

[0024] The tail values ​​of the pixel values ​​of at least part of the pixels on multiple channels are replaced by the values ​​of the sampling points in the sampling point sequence.

[0025] In some embodiments, the synchronization description information is used to describe the synchronization relationship between the image area in the image and the audio content in the audio;

[0026] According to the synchronization description information, a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence is established to obtain audio synchronization information, including:

[0027] According to the synchronization description information, a corresponding relationship between the image area in the image and the sampling points in the sampling point sequence is established to obtain the audio synchronization information.

[0028] In some embodiments, the synchronization description information is used to describe the synchronization relationship between the operation sequence of the plurality of image regions in the image and the audio content in the audio;

[0029] According to the synchronization description information, a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence is established to obtain audio synchronization information, including:

[0030] According to the synchronization description information, a corresponding relationship between an operation sequence of a plurality of image blocks in the image and sampling points in the sampling point sequence is established to obtain the audio synchronization information.

[0031] In some embodiments, before sampling the audio to obtain a sampling point sequence, the method further includes:

[0032] Perform audio content deduplication processing on the audio.

[0033] In a third aspect, an embodiment of the present application provides a multimedia file playback device, including:

[0034] A processing module, configured to display an image in the multimedia file in response to an opening operation of the multimedia file, and extract a sampling point sequence of the steganographic audio from pixel information of the image;

[0035] A first determining module, configured to determine operation information in response to a user operation on the image;

[0036] A second determination module, configured to determine, from the sampling point sequence, a sampling point corresponding to the operation information according to audio synchronization information in the multimedia file, wherein the audio synchronization information is used to indicate a correspondence between the operation information and the sampling points in the sampling point sequence;

[0037] The playing module is used to play the audio according to the determined sampling points.

[0038] In a fourth aspect, an embodiment of the present application provides a device for generating a multimedia file, including:

[0039] An acquisition module, used to acquire an image, audio and synchronization description information, wherein the synchronization description information is used to describe the synchronization relationship between the operation information of the image and the audio content in the audio;

[0040] An audio processing module, used for sampling the audio to obtain a sampling point sequence, and steganographically writing the sampling points in the sampling point sequence into the pixel information of the image;

[0041] A synchronization module, used to establish a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence according to the synchronization description information, so as to obtain audio synchronization information;

[0042] A generation module is used to generate a multimedia file in a multimedia file format based on the pixel information of the image after steganography and the audio synchronization information.

[0043] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0044] The memory stores a computer program executable by at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute any of the above-mentioned methods for playing multimedia files or methods for generating multimedia files.

[0045] In a sixth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned methods for playing multimedia files or methods for generating multimedia files.

[0046] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for playing multimedia files or methods for generating multimedia files.

[0047] In the embodiment of the present application, in response to the opening operation of the multimedia file, the image in the multimedia file is displayed, the sampling point sequence of the steganographic audio is extracted from the pixel information of the image, in response to the user operation on the image, the operation information is determined, and the sampling point corresponding to the operation information is determined from the sampling point sequence according to the audio synchronization information in the multimedia file, and the audio synchronization information is used to indicate the corresponding relationship between the operation information and the sampling points in the sampling point sequence, and then the audio is played according to the determined sampling points. In this way, the multimedia file formed by the combination of image and audio contains audio synchronization information, and with the help of the audio synchronization information, the synchronous playback mode of the image and audio can be refined to the operation information of the image and the sampling points of the audio. The refined synchronous playback mode is conducive to improving the playback flexibility of the multimedia file, and the audio is sampled and steganographically written in the pixel information of the image, which does not change the size of the image, so the size of the multimedia file is relatively small. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 A schematic diagram of an application scenario of a method for playing multimedia files provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of a method for generating a multimedia file provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of an image provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of another image provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of another image provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the structure of a multimedia file playback device provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of the structure of a device for generating a multimedia file provided in an embodiment of the present application;

[0056] Figure 8 A schematic diagram of the hardware structure of an electronic device for implementing a method for playing a multimedia file or a method for generating a multimedia file provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to solve the problem in the related art that when combining images and audio, the playback mode of the combined multimedia file is single and the multimedia file is relatively large, an embodiment of the present application provides a method for playing and generating multimedia files.

[0058] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0059] See also Figure 1 , Figure 1 A flowchart of a method for playing a multimedia file provided in an embodiment of the present application includes the following steps.

[0060] In step 101, in response to an opening operation of a multimedia file, an image in the multimedia file is displayed, and a sampling point sequence of the steganographic audio is extracted from pixel information of the image.

[0061] Among them, the pixel information of the image may include the pixel values ​​of each pixel on multiple channels. Therefore, when extracting the sampling point sequence of the steganographic audio from the pixel information of the image, the tail values ​​of the pixel values ​​of at least some pixels on multiple channels can be extracted, and then the extracted values ​​are combined to obtain the sampling point sequence.

[0062] That is to say, the audio is hidden in the image, which does not change the image size, so the size of the multimedia file is also relatively small.

[0063] In step 102, operation information is determined in response to a user operation on an image.

[0064] Among them, user operations include clicking, sliding, double-clicking, long pressing, etc., and operation information includes which image blocks in the image are operated, the operation order of these image blocks, etc.

[0065] In step 103, the sampling points corresponding to the operation information are determined from the sampling point sequence according to the audio synchronization information in the multimedia file, and the audio synchronization information is used to indicate the corresponding relationship between the operation information and the sampling points in the sampling point sequence.

[0066] In some embodiments, the audio synchronization information is used to indicate the correspondence between the image region in the image and the sampling points in the sampling point sequence, and the operation information includes the target image region in the image to be operated. In this case, the sampling points corresponding to the operation information are determined from the sampling point sequence according to the audio synchronization information in the multimedia file, that is, the sampling points corresponding to the target image region in the sampling point sequence are determined as the sampling points corresponding to the operation information according to the audio synchronization information.

[0067] In some embodiments, the audio synchronization information is used to indicate the correspondence between the operation sequence of multiple image regions in the image and the sampling points in the sampling point sequence, and the operation information includes the operation sequence of multiple target image regions in the image. In this case, the sampling points corresponding to the operation information are determined from the sampling point sequence according to the audio synchronization information in the multimedia file, that is, the sampling points corresponding to the operation sequence of the multiple target image regions in the sampling point sequence are determined as the sampling points corresponding to the operation information according to the audio synchronization information.

[0068] In step 104, audio playback is performed according to the determined sampling points.

[0069] That is, according to the audio playback format, the audio of the determined sampling point is restored and then played.

[0070] In this way, the audio hidden in the image can be played synchronously when the user operates the image, providing a more flexible way of synchronously playing images and audio.

[0071] See also Figure 2 , Figure 2 A flowchart of a method for generating a multimedia file provided in an embodiment of the present application,

[0072] In step 201, an image, audio and synchronization description information are obtained, where the synchronization description information is used to describe the synchronization relationship between the operation information of the image and the audio content in the audio.

[0073] The synchronous description information includes, for example, what audio content is played when clicking each image area of ​​the image, and what audio content is played when swiping across multiple image areas in the image in what order.

[0074] In step 202, the audio is sampled to obtain a sampling point sequence, and the sampling points in the sampling point sequence are steganographically written in the pixel information of the image.

[0075] In some embodiments, before sampling the audio to obtain a sampling point sequence, the audio content may be deduplicated. This can reduce the amount of audio data to be processed and the amount of audio data to be steganographically written into the image, thereby increasing the speed of generating multimedia files.

[0076] Generally, the pixel information of an image may include the pixel values ​​of each pixel on multiple channels. Therefore, when the sampling points in the sampling point sequence are steganographically written in the pixel information of the image, the tail values ​​of the pixel values ​​of at least some pixels on multiple channels may be replaced by the values ​​of the sampling points in the sampling point sequence.

[0077] In step 203, according to the synchronization description information, a synchronization relationship between the image operation information and the sampling points in the sampling point sequence is established to obtain audio synchronization information.

[0078] In some embodiments, the synchronization description information is used to describe the synchronization relationship between the image area in the image and the audio content in the audio. According to the synchronization description information, the synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence is established to obtain the audio synchronization information. That is, according to the synchronization description information, the corresponding relationship between the image area in the image and the sampling points in the sampling point sequence is established to obtain the audio synchronization information.

[0079] In some embodiments, the synchronization description information is used to describe the synchronization relationship between the operation order of multiple image areas in the image and the audio content in the audio. According to the synchronization description information, the synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence is established to obtain the audio synchronization information. That is, according to the synchronization description information, the corresponding relationship between the operation order of multiple image blocks in the image and the sampling points in the sampling point sequence is established to obtain the audio synchronization information.

[0080] In step 204, a multimedia file is generated in a multimedia file format based on the pixel information of the steganographic image and the audio synchronization information.

[0081] The solution of the embodiment of the present application is introduced below with reference to specific examples.

[0082] See Table 1, which shows a format of a multimedia file composed of an image and audio. Assuming that this format is called the Audio-Enhanced Image Format (AEIF) format, the multimedia file generated according to Table 1 is in the AEIF format.

[0083] Table 1

[0084]

[0085] The following is an introduction to the solutions of the embodiments of the present application.

[0086] Scenario 1: The sender sends images and audio to the receiver at the same time.

[0087] Assuming that the image to be sent is a 24-bit RGB image of 3*2 pixels, and the audio is a 1-second audio with a sampling rate of 2Hz, 8-bit mono, and a length of 1 second, the least significant bit (LSB) steganography can be used to embed the audio data into the image data, and the starting pixel for audio and image synchronization is the first pixel, the offset is three pixels, and the mapping method is linear. Finally, according to the file structure shown in Table 1, the data is assembled to generate a multimedia file in AEIF format.

[0088] The above process is described in detail below.

[0089] The processing at the sending end includes the following steps.

[0090] 1. Refer to Table 2 to define the values ​​corresponding to the eight field types.

[0091] Table 2

[0092] type value F_HEAD 0x01 IMG_HEAD 0x02 IMG_DATA 0x03 IMG_END 0x04 AUD_HEAD 0x05 AUD_END 0x06 IA_SYNC 0x07 F_END 0x08

[0093] 2. Define a multimedia file with the type of AEIF, version 1.0, file size of 1024 bytes, using LSB steganography method, and creation time of 2025-01-01 00:00:00.

[0094] The structure of the F_HEAD file header is shown in Table 3.

[0095] Table 3

[0096]

[0097]

[0098] The data in the F_HEAD file header is shown in Table 4.

[0099] Table 4

[0100] Fields content describe File Identification 0x41454946 Type:AEIF File Version 0x0100 Version number 1.0 File size 0x0400 The total size is 1024 bytes Steganography Methods 0x01 Steganography method is LSB Creation time 0x0000018CC251F400 The creation time is 2024-01-01 00:00:00

[0101] 3. Define the image, a 3*2 pixel 24-bit RGB image, where each RGB channel is represented by 8 bits.

[0102] The structure of the IMG_HEAD image header is shown in Table 5.

[0103] Table 5

[0104]

[0105] The data in the IMG_HEAD image header is shown in Table 6.

[0106] Table 6

[0107] Fields content describe width 0x00000003 Image width is 3 pixels high 0x00000002 Image height is 2 pixels Color Type 0x01 Color type is RGB Bit Depth 0x08 Each color channel depth is 8 bits Compression method 0x01 A compression method Filter Method 0x01 A filtering method Interleaving Method 0x01 A staggered approach

[0108] The image data (without embedded audio data) is shown in Table 7.

[0109] Table 7

[0110]

[0111] In Table 7, the numbers in brackets are the decimal values ​​of the binary numbers outside the brackets.

[0112] 4. Define audio, with a sampling rate of 2Hz, 8-bit mono, and a length of 1 second. The structure of the AUD_HEAD audio header is shown in Table 8.

[0113] Table 8

[0114]

[0115] The data in the AUD_HEAD audio header is shown in Table 9.

[0116] Table 9

[0117] Fields content describe Sampling rate 0x00000002 Sampling rate 2Hz Number of channels 0x00000001 The number of channels is 1 Sample bit depth 0x0008 Each sample point has a bit depth of 8 bits Audio Length 0x00000001 The audio length is 1 second Compression method 0x01 A compression method

[0118] The audio data is specifically shown in Table 10.

[0119] Table 10

[0120]

[0121] 5. According to the structure information of the F_HEAD file header, use LSB steganography to embed the audio data in step 4 into the image data in step 3.

[0122] The structure of IMG_DATA is shown in Table 11.

[0123] Table 11

[0124]

[0125] Among them, the first 3 bits of the first sampling point of the audio in step 4 are respectively embedded into the least significant bits of the RGB channels of the first pixel point of the image in step 3, as follows:

[0126] The first 3 bits of the first audio sampling point: 011,

[0127] The first pixel of the image:

[0128] R:11001001(201)

[0129] G:11011000(216)

[0130] B:11101100(236),

[0131] Then, embed the first bit (0) of the first audio sample point into the least significant bit of the red channel:

[0132] R:11001000(200),

[0133] Embed the second bit (1) of the first audio sample into the least significant bit of the green channel:

[0134] G:11011001(217),

[0135] Embed the third bit (1) of the first audio sample into the least significant bit of the blue channel:

[0136] B:11101101(237).

[0137] Similarly, the remaining 5 bits of the first audio sampling point and the 8 bits of the second audio sampling point are embedded into the RGB channels of the second to fifth pixels and the R channel of the sixth pixel of the image, respectively. Finally, all the audio data is embedded into the image.

[0138] The data in IMG_DATA is shown in Table 12.

[0139] Table 12

[0140]

[0141] In Table 12, the numbers in brackets are the decimal values ​​of the binary numbers outside the brackets, and the underlined numbers in Table 12 are the values ​​of the embedded audio sampling points.

[0142] 6. Define the synchronization information of audio and image. The starting pixel is the first pixel, the synchronization offset is three pixels, and the mapping method is linear.

[0143] The structure of IA_SYNC is shown in Table 13.

[0144] Table 13

[0145]

[0146]

[0147] The data in IA_SYNC is shown in Table 14.

[0148] Table 14

[0149]

[0150] 7. According to the IA_SYNC information, each sampling point of the audio in step 4 is mapped to a pixel point of the image in which the audio is embedded in step 5.

[0151] The details are as follows:

[0152] Starting pixel: the first pixel;

[0153] Synchronization offset: three pixels;

[0154] Mapping method: linear, that is, the first audio sampling point corresponds to the first pixel of the image, and the second audio sampling point corresponds to the fourth pixel of the image.

[0155] In this way, when playing, when the first pixel of the image is displayed, the first sampling point of the audio will be played, and when the fourth pixel of the image is displayed, the second sampling point of the audio will be played.

[0156] 8. IMG_END, AUD_END, and F_END do not require data.

[0157] The structure of IMG_END is shown in Table 15.

[0158] Table 15

[0159]

[0160]

[0161] The structure of AUD_END is shown in Table 16.

[0162] Table 16

[0163]

[0164] The structure of F_END is shown in Table 17.

[0165] Table 17

[0166]

[0167] 9. According to Table 1, assemble F_HEAD, IMG_HEAD, IMG_DATA, IMG_END, AUD_HEAD, AUD_END, IA_SYNC and F_END in order to obtain a multimedia file in AEIF format. 10. Send this multimedia file in AEIF format to the receiving end through the network.

[0168] After receiving the multimedia file in AEIF format, the receiving end decodes it in sequence, wherein each data block (one data block for each field type) is decoded according to the structure, and the fields are size, type, data, and CRC in sequence.

[0169] The processing at the receiving end includes the following steps.

[0170] 1. Decode the first data block.

[0171] The structure of the first data block is shown in Table 18.

[0172] Table 18

[0173] Fields content size For example: 0x00000400 type 0x01 data - CRC For example: 0x417CD0E9

[0174] The data block type is 0x01, which is F_HEAD, indicating that it can be decoded according to the F_HEAD type. The fields are file identifier, file version, file size, steganography method, and creation time. The size field indicates the length of the decoded data block, and finally the integrity of the data block is verified through the CRC field; if the verification passes, the decoding continues, otherwise, the decoding fails.

[0175] The data in the first data block is shown in Table 19.

[0176] Table 19

[0177]

[0178]

[0179] Decoding result: The multimedia file is of type AEIF, version 1.0, file size 1024 bytes, uses LSB steganography method, and creation time is 2024-01-01 00:00:00.

[0180] 2. Decode the second data block.

[0181] The structure of the second data block is shown in Table 20.

[0182] Table 20

[0183] Fields content size For example: 0x00000400 type 0x02 data - CRC For example: 0x4C22B152

[0184] The data block type is 0x02, that is, IMG_HEAD, which means decoding according to the IMG_HEAD type. The fields are width, height, color type, bit depth, compression method, filtering method, and interleaving method. The size field indicates the length of the decoded data block, and finally the integrity of the data block is verified through the CRC field; if the verification passes, the decoding continues, otherwise, the decoding fails.

[0185] The data in the second data block is shown in Table 21.

[0186] Table 21

[0187]

[0188]

[0189] Decoding result: The image in the multimedia file is 3*2 pixels 24-bit RGB, where each RGB channel uses 8 bits, using a certain compression method, a certain filtering method, and a certain interleaving method.

[0190] 3. Decode the third data block.

[0191] The structure of the third data block is shown in Table 22.

[0192] Table 22

[0193] Fields content size For example: 0x00000400 type 0x03 data - CRC For example: 0x87D6E02D

[0194] The data block type is 0x03, that is, IMG_DATA, which means decoding according to the IMG_DATA type. The field is image data. The size field indicates the length of the decoded data block, and finally the integrity of the data block is verified through the CRC field; if the verification passes, the decoding continues, otherwise, the decoding fails.

[0195] The data in the third data block is shown in Table 23.

[0196] Table 23

[0197]

[0198]

[0199] Decoding result: The image embedded with audio in the multimedia file is in 3x2 pixel, 24-bit RGB format, and is displayed after specific compression, filtering and interleaving methods (the specific width, height, color type, bit depth, and compression, filtering and interleaving methods can be obtained through the decoding information of IMG_HEAD type).

[0200] 4. Decode the fourth data block.

[0201] The structure of the fourth data block is shown in Table 24.

[0202] Table 24

[0203] Fields content size For example: 0x00000000 type 0x04 data No data CRC For example: 0x4066F07E

[0204] The data block type is 0x04, that is, IMG_END type, indicating the end of image decoding. The size field indicates the length of the decoded data block. Finally, the integrity of the data block is verified through the CRC field; if the verification passes, the decoding continues, otherwise, the decoding fails.

[0205] 5. The fifth data block is decoded.

[0206] The structure of the fifth data block is shown in Table 25.

[0207] Table 25

[0208]

[0209]

[0210] The data block type is 0x05, which is AUD_HEAD type, indicating that the decoding is performed according to the AUD_HEAD type. The fields are sampling rate, number of channels, sampling bit depth, audio length, and compression method. The size field indicates the length of the decoded data block. Finally, the integrity of the data block is verified through the CRC field; if the verification passes, the decoding continues, otherwise the decoding fails.

[0211] The data in the fifth data block is shown in Table 26.

[0212] Table 26

[0213] Fields content Sampling rate 0x00000002 Number of channels 0x00000001 Sample bit depth 0x0008 Audio Length 0x00000001 Compression method 0x01

[0214] Decoding result: The audio embedded in the image data in the multimedia file has a sampling rate of 2Hz, 8-bit mono, a length of 1 second, and uses a certain compression method.

[0215] At the same time, the decoding result in step 1 shows that the audio is embedded in the image data using the LSB steganography method. Therefore, based on the above audio information and the image data in step 3, the audio data can be extracted as shown in Table 27.

[0216] Table 27

[0217]

[0218]

[0219] 6. The sixth data block is decoded.

[0220] The structure of the sixth data block is shown in Table 28.

[0221] Table 28

[0222] Fields content size For example: 0x00000000 type 0x06 data No data CRC For example: 0xD2C99B66

[0223] The data block type is 0x06, which is AUD_END type, indicating the end of audio decoding. The size field indicates the length of the decoded data block. Finally, the integrity of the data block is verified through the CRC field; if the verification passes, decoding continues, otherwise, decoding fails.

[0224] 7. The seventh data block is decoded.

[0225] The structure of the seventh data block is shown in Table 29.

[0226] Table 29

[0227] Fields content size For example: 0x00000400 type 0x07 data - CRC For example: 0x572F6516

[0228] The data block type is 0x07, which is the IA_SYNC type, indicating that the decoding is performed according to the IA_SYNC type. The fields are the starting pixel, synchronization offset, and mapping mode. The size field indicates the length of the decoded data block. Finally, the integrity of the data block is verified through the CRC field; if the verification passes, the decoding continues, otherwise the decoding fails.

[0229] The data in the seventh data block is shown in Table 30.

[0230] Table 30

[0231] Fields content describe Start Pixel 0x00000001 The start of audio and image synchronization is the first pixel Synchronization offset 0x00000003 The offset of audio and image synchronization is three pixels Mapping method 0x01 The mapping method of audio image pixels is linear

[0232] Decoding result: The audio and image synchronization information in the multimedia file is that the starting pixel is the first pixel, the synchronization offset is three pixels, and the mapping method adopts linear mapping.

[0233] At the same time, according to the above decoding results, each sampling point of the audio in step 5 is mapped to the pixel point of the image in step 3, as follows:

[0234] Starting pixel: the first pixel;

[0235] Synchronization offset: three pixels;

[0236] Mapping method: linear, that is, the first audio sampling point corresponds to the first pixel of the image, and the second audio sampling point corresponds to the fourth pixel of the image.

[0237] In this way, when the first pixel of the image is displayed, the first sample point of the audio will be played, and when the fourth pixel of the image is displayed, the second sample point of the audio will be played.

[0238] 8. The eighth data block is decoded.

[0239] The structure of the eighth data block is shown in Table 31.

[0240] Table 31

[0241]

[0242]

[0243] The data block type is 0x08, that is, F_END type, indicating the end of the decoding of the entire multimedia file. The size field indicates the length of the decoded data block. Finally, the integrity of the data block is verified through the CRC field; if the verification passes, the decoding continues, otherwise, the decoding fails.

[0244] Scenario 2: Different audio content is played when different areas of the image are operated.

[0245] See also Figure 3 , Figure 3 A schematic diagram of an image provided for an embodiment of the present application, in which the image is divided into 9 areas, each area having different image content. In the order of rows first and then columns, the audio corresponding to image area 1 to image area 9 are: cabbage, celery, spinach, leek, carrot, white radish, cucumber, pumpkin, and winter melon.

[0246] In specific implementation, the audio content in the audio can be first de-duplicated to obtain the audio file: white celery, spinach, leek, red carrot, yellow south wax gourd. Then, the words in the image area and the audio file are re-corresponded as follows:

[0247] Image area 1 -> the 1st word "white", the 5th word "celery";

[0248] Image area 2 -> the 2nd word "celery", the 5th word "celery";

[0249] Image area 3 -> the 3rd word "spinach", the 5th word "celery";

[0250] Image area 4 -> the 4th word "leek", the 5th word "celery";

[0251] Image area 5 -> the 6th word "red", the 7th word "carrot", the 8th word "radish";

[0252] Image area 6 -> the 1st word "white", the 7th word "carrot", the 8th word "radish";

[0253] Image area 7 -> the 9th word "yellow", the 12th word "wax gourd";

[0254] Image area 8 -> the 10th word "south", the 12th word "wax gourd";

[0255] Image area 9 -> the 11th word "winter", the 12th word "wax gourd".

[0256] Then, this audio file is embedded into the image using steganography, and the above mapping method is written into the data block corresponding to IA_SYNC.

[0257] In this way, through audio content de-duplication and steganography, the size of the entire multimedia file can be greatly reduced. And during playback, as long as the user clicks on a certain image area, the audio content corresponding to the image area can be played, and the entire image area can be used as the trigger area for audio playback, which is convenient for user operation.

[0258] Scenario 3: When different areas of the image are operated in a given order, the corresponding audio content is played.

[0259] See Figure 4 Figure 4 , which is a schematic diagram of an image provided by an embodiment of the present application. The image is divided into n image areas according to the stroke order of the stroke "丶" (from top to bottom), and the 1st, 2nd, 3rd... nth image areas are respectively defined. Then, the n image areas connected in stroke order are mapped to the audio file "dot".

[0260] ​Subsequently, if the user draws from top to bottom in the image in the order of strokes, the audio "dot" will be triggered to play. If the user does not draw from top to bottom in the image in the order of strokes, the audio will not be triggered to play.

[0261] See also Figure 5 , Figure 5 A schematic diagram of another image provided for an embodiment of the present application, in which the image is divided into n image areas according to the stroke order of the stroke “—” (from left to right), and 1, 2, 3…n image areas are defined respectively, and then the n image areas connected in stroke order are mapped to the audio file “Horizontal”.

[0262] Subsequently, if the user draws from left to right in the image according to the stroke order, the audio "horizontal" will be triggered to play. If the user does not draw from top to bottom in the image according to the stroke order, the audio will not be triggered to play.

[0263] In this way, it supports mapping multiple areas in the image to one audio, and at the same time, multiple image areas are connected and mapped to the audio in a certain order, which can achieve the purpose of learning the stroke order of Chinese characters.

[0264] Based on the same technical concept, the embodiment of the present application also provides a multimedia file playback device. The principle of solving the problem by the multimedia file playback device is similar to that of the multimedia file playback method mentioned above. Therefore, the implementation of the multimedia file playback device can refer to the implementation of the multimedia file playback method, and the repeated parts will not be repeated.

[0265] Figure 6 A schematic diagram of a multimedia file playback device provided in an embodiment of the present application includes:

[0266] The processing module 601 is used for displaying an image in the multimedia file in response to an opening operation of the multimedia file, and extracting a sampling point sequence of the steganographic audio from pixel information of the image;

[0267] A first determining module 602, configured to determine operation information in response to a user operation on the image;

[0268] A second determination module 603 is used to determine the sampling point corresponding to the operation information from the sampling point sequence according to the audio synchronization information in the multimedia file, wherein the audio synchronization information is used to indicate the correspondence between the operation information and the sampling points in the sampling point sequence;

[0269] The playing module 604 is used to play the audio according to the determined sampling points.

[0270] In some embodiments, the pixel information of the image includes pixel values ​​of each pixel on multiple channels, and the processing module 601 is specifically used to:

[0271] Extracting tail values ​​of pixel values ​​of at least some pixels on multiple channels;

[0272] The extracted values ​​are combined to obtain the sampling point sequence.

[0273] In some embodiments, the audio synchronization information is used to indicate a correspondence between an image region in the image and sampling points in the sampling point sequence, and the operation information includes a target image region in the image to be operated;

[0274] The second determination module 603 is specifically configured to determine, according to the audio synchronization information, the sampling point corresponding to the target image area in the sampling point sequence as the sampling point corresponding to the operation information.

[0275] In some embodiments, the audio synchronization information is used to indicate the corresponding relationship between the operation sequence of the plurality of image regions in the image and the sampling points in the sampling point sequence; the operation information includes the operation sequence of the plurality of target image regions in the image;

[0276] The second determination module 603 is specifically configured to determine, according to the audio synchronization information, sampling points corresponding to the operation sequences of the plurality of target image regions in the sampling point sequence as sampling points corresponding to the operation information.

[0277] Based on the same technical concept, the embodiment of the present application also provides a device for generating a multimedia file. The principle of solving the problem by the device for generating a multimedia file is similar to that of the above-mentioned method for generating a multimedia file. Therefore, the implementation of the device for generating a multimedia file can refer to the implementation of the method for generating a multimedia file, and the repeated parts will not be repeated.

[0278] Figure 7 A schematic diagram of a structure of a multimedia file generation device provided in an embodiment of the present application includes:

[0279] An acquisition module 701 is used to acquire an image, audio, and synchronization description information, wherein the synchronization description information is used to describe the synchronization relationship between the operation information of the image and the audio content in the audio;

[0280] An audio processing module 702, configured to sample the audio to obtain a sampling point sequence, and to steganographically write the sampling points in the sampling point sequence into the pixel information of the image;

[0281] A synchronization module 703, configured to establish a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence according to the synchronization description information, so as to obtain audio synchronization information;

[0282] The generating module 704 is used to generate a multimedia file in a multimedia file format based on the pixel information of the image after steganography and the audio synchronization information.

[0283] In some embodiments, the pixel information of the image includes pixel values ​​of each pixel on multiple channels, and the audio processing module 702 is specifically configured to:

[0284] The tail values ​​of the pixel values ​​of at least part of the pixels on multiple channels are replaced by the values ​​of the sampling points in the sampling point sequence.

[0285] In some embodiments, the synchronization description information is used to describe the synchronization relationship between the image area in the image and the audio content in the audio;

[0286] The synchronization module 703 is specifically configured to establish a correspondence between an image region in the image and sampling points in the sampling point sequence according to the synchronization description information, so as to obtain the audio synchronization information.

[0287] In some embodiments, the synchronization description information is used to describe the synchronization relationship between the operation sequence of the plurality of image regions in the image and the audio content in the audio;

[0288] The synchronization module 703 is specifically configured to establish a correspondence between an operation sequence of multiple image blocks in the image and sampling points in the sampling point sequence according to the synchronization description information, so as to obtain the audio synchronization information.

[0289] In some embodiments, the audio processing module 702 is further configured to:

[0290] Before sampling the audio to obtain a sampling point sequence, audio content deduplication processing is performed on the audio.

[0291] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated in a processor, or may exist physically separately, or two or more modules may be integrated in one module. The coupling between the modules may be achieved through some interfaces, which are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0292] After introducing the multimedia file playback and generation method and related devices according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0293] Refer to the following Figure 8 The electronic device 130 implemented according to this embodiment of the present application is described. Figure 8 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0294] like Figure 8 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0295] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0296] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0297] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0298] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0299] In an exemplary embodiment, a storage medium is also provided, and when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned methods for playing multimedia files or methods for generating multimedia files. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0300] In an exemplary embodiment, the electronic device of the present application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor can execute the steps of any multimedia file playback method or multimedia file generation method provided in the embodiments of the present application.

[0301] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed by an electronic device, the electronic device can implement any exemplary method provided in the present application.

[0302] Furthermore, the computer program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0303] The program product for playing and generating multimedia files in the embodiment of the present application can adopt a CD-ROM and include program code, and can be run on a computing device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, and the program can be used by or in combination with an instruction execution system, an apparatus or a device.

[0304] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0305] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0306] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0307] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0308] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0309] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0310] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0311] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0312] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0313] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0314] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also includes these modifications and variations.

Claims

1. A method for playing a multimedia file, characterized in that: include: In response to an opening operation of a multimedia file, an image in the multimedia file is displayed, and a sampling point sequence of the steganographic audio is extracted from pixel information of the image; determining operation information in response to a user operation on the image; Determining, from the sampling point sequence, a sampling point corresponding to the operation information according to the audio synchronization information in the multimedia file, wherein the audio synchronization information is used to indicate a corresponding relationship between the operation information and the sampling points in the sampling point sequence; The audio is played according to the determined sampling points.

2. The method according to claim 1, characterized in that The pixel information of the image includes pixel values ​​of each pixel on multiple channels, and extracting a sampling point sequence of the steganographic audio from the pixel information of the image includes: Extracting tail values ​​of pixel values ​​of at least some pixels on multiple channels; The extracted values ​​are combined to obtain the sampling point sequence.

3. The method according to claim 1 or 2, characterized in that The audio synchronization information is used to indicate the corresponding relationship between the image area in the image and the sampling points in the sampling point sequence, and the operation information includes the target image area in the image to be operated; Determining the sampling point corresponding to the operation information from the sampling point sequence according to the audio synchronization information in the multimedia file includes: According to the audio synchronization information, the sampling point corresponding to the target image area in the sampling point sequence is determined as the sampling point corresponding to the operation information.

4. The method according to claim 1 or 2, characterized in that: The audio synchronization information is used to indicate the corresponding relationship between the operation sequence of the multiple image regions in the image and the sampling points in the sampling point sequence; the operation information includes the operation sequence of the multiple target image regions in the image; Determining the sampling point corresponding to the operation information from the sampling point sequence according to the audio synchronization information in the multimedia file includes: According to the audio synchronization information, sampling points corresponding to the operation order of the plurality of target image regions in the sampling point sequence are determined as sampling points corresponding to the operation information.

5. A method for generating a multimedia file, characterized in that: include: Acquire an image, audio, and synchronization description information, wherein the synchronization description information is used to describe the synchronization relationship between the operation information of the image and the audio content in the audio; Sampling the audio to obtain a sampling point sequence, and steganographically writing the sampling points in the sampling point sequence into the pixel information of the image; According to the synchronization description information, a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence is established to obtain audio synchronization information; Based on the pixel information of the steganographic image and the audio synchronization information, a multimedia file is generated in a multimedia file format.

6. The method according to claim 5, characterized in that The pixel information of the image includes pixel values ​​of each pixel on multiple channels, and the sampling points in the sampling point sequence are hidden in the pixel information of the image, including: The tail values ​​of the pixel values ​​of at least part of the pixels on multiple channels are replaced by the values ​​of the sampling points in the sampling point sequence.

7. The method according to claim 5 or 6, characterized in that The synchronization description information is used to describe the synchronization relationship between the image area in the image and the audio content in the audio; According to the synchronization description information, a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence is established to obtain audio synchronization information, including: According to the synchronization description information, a corresponding relationship between the image area in the image and the sampling points in the sampling point sequence is established to obtain the audio synchronization information.

8. The method according to claim 5 or 6, characterized in that The synchronization description information is used to describe the synchronization relationship between the operation sequence of the plurality of image regions in the image and the audio content in the audio; According to the synchronization description information, a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence is established to obtain audio synchronization information, including: According to the synchronization description information, a corresponding relationship between an operation sequence of a plurality of image blocks in the image and sampling points in the sampling point sequence is established to obtain the audio synchronization information.

9. The method according to claim 5, characterized in that Before sampling the audio to obtain a sampling point sequence, the method further includes: Perform audio content deduplication processing on the audio.

10. A multimedia file playback device, characterized in that: include: A processing module, configured to display an image in the multimedia file in response to an opening operation of the multimedia file, and extract a sampling point sequence of the steganographic audio from pixel information of the image; A first determining module, configured to determine operation information in response to a user operation on the image; A second determination module, configured to determine, from the sampling point sequence, a sampling point corresponding to the operation information according to audio synchronization information in the multimedia file, wherein the audio synchronization information is used to indicate a correspondence between the operation information and the sampling points in the sampling point sequence; The playing module is used to play the audio according to the determined sampling points.

11. A device for generating a multimedia file, characterized in that: include: An acquisition module, used to acquire an image, audio and synchronization description information, wherein the synchronization description information is used to describe the synchronization relationship between the operation information of the image and the audio content in the audio; An audio processing module, used for sampling the audio to obtain a sampling point sequence, and steganographically writing the sampling points in the sampling point sequence into the pixel information of the image; A synchronization module, used to establish a synchronization relationship between the operation information of the image and the sampling points in the sampling point sequence according to the synchronization description information, so as to obtain audio synchronization information; A generation module is used to generate a multimedia file in a multimedia file format based on the pixel information of the image after steganography and the audio synchronization information.

12. An electronic device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 9.

13. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the method according to any one of claims 1 to 9.

14. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images