Time code acquisition method, time code acquisition device, electronic equipment and program product

Abstract

The invention is suitable for the technical field of multimedia, and provides a time code acquisition method, a time code acquisition device, electronic equipment and a program product. The method comprises the following steps: acquiring a plurality of materials; under the condition that a first material exists in the multiple materials and the first material contains audio data, if a second material exists in the multiple materials, determining a time code of the first material based on a time code of the second material; wherein the first material is a material which does not contain the time code or the time code is wrong, the second material is a material which contains the audio data and the time code, and the audio waveform of the contained audio data is matched with the audio waveform of the audio data of the first material. According to the method and the device, the time code of the material which does not contain the time code or has the wrong time code can be obtained, so that the availability of the material is improved.

Description

Technical Field

[0001] The present application belongs to the field of multimedia technology, and in particular relates to a time code acquisition method, a time code acquisition device, an electronic device, and a program product. Background Art

[0002] When processing multi-camera footage synchronously, existing technologies primarily rely on timecodes to achieve this. Timecodes are typically generated and transmitted to each capture device by a timecode synchronization device. However, timecode synchronization devices are subject to downtime, and timecode transmission is typically wireless, resulting in low communication reliability. These issues can result in footage lacking timecode or containing incorrect timecodes, reducing the usability of the footage. Summary of the Invention

[0003] The embodiments of the present application provide a time code acquisition method, a time code acquisition device, an electronic device, and a program product, which can acquire the time code of materials that do not contain time codes or have incorrect time codes, thereby improving the usability of materials that do not contain time codes or have incorrect time codes.

[0004] In a first aspect, an embodiment of the present application provides a time code acquisition method, comprising:

[0005] Get multiple materials;

[0006] If a first material exists among the plurality of materials and the first material includes audio data, if a second material exists among the plurality of materials, determining the time code of the first material based on the time code of the second material;

[0007] The first material is a material that does not contain a time code or has an incorrect time code, and the second material is a material that contains audio data and a time code, and the audio waveform of the audio data contained therein matches the audio waveform of the audio data of the first material.

[0008] In an embodiment of the present application, after obtaining multiple materials, if there is a first material among the multiple materials and the first material contains audio data, it can be determined whether there is a second material among the multiple materials. The second material is a material that contains audio data and time code and the audio waveform of the audio data contained therein matches the audio waveform of the audio data of the first material. Since the audio data of the two materials with waveform matching are the same or relatively similar, the relationship between the time codes of the two materials can be determined based on the waveform matching. Therefore, if there is a second material among the multiple materials, the time code of the first material can be determined based on the time code of the second material, that is, the time code of the material that does not contain a time code or has an incorrect time code can be obtained based on the time code of the second material, thereby improving the availability of the material that does not contain a time code or has an incorrect time code.

[0009] In a second aspect, an embodiment of the present application provides a time code acquisition device, comprising:

[0010] Material acquisition module, used to acquire multiple materials;

[0011] a time code determining module configured to, when a first material exists among the plurality of materials and the first material includes audio data, determine a time code of the first material based on a time code of the second material if a second material exists among the plurality of materials;

[0012] The first material is a material that does not contain a time code or has an incorrect time code, and the second material is a material that contains audio data and a time code, and the audio waveform of the audio data contained therein matches the audio waveform of the audio data of the first material.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements a time code acquisition method as described in any one of the first aspects above.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the time code acquisition method as described in the first aspect above is implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is run, the time code acquisition method as described in any one of the above-mentioned first aspects is executed.

[0016] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is a flow chart of a time code acquisition method provided by an embodiment of the present application;

[0019] Figure 2 This is another flowchart of the time code acquisition method provided by an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the structure of the time code acquisition device provided in an embodiment of the present application;

[0021] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0023] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0025] The time code acquisition method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0026] See also Figure 1 , Figure 1 The following is a flow chart of a method for obtaining a time code according to an embodiment of the present application. As an example and not a limitation, the method includes the following steps:

[0027] Step 101: Acquire multiple materials.

[0028] The above-mentioned multiple materials (i.e., at least two materials) may include at least one of video materials and audio materials. Video materials may or may not include audio data, which is not limited in this application. Audio materials generally include audio data but do not include images or pictures.

[0029] As an example and not a limitation, a time code acquisition application is installed on an electronic device. A user can import multiple materials into the time code acquisition application, thereby using the time code acquisition application to determine the time code of the first material among the multiple materials. Based on this, materials of the same scene shot or recorded by different material acquisition devices can be aligned in time to achieve multi-camera material synchronization. The above-mentioned time code acquisition application is program code or software that implements the time code acquisition method provided in the embodiments of the present application.

[0030] Optionally, the above-mentioned material collection equipment includes but is not limited to video equipment, recording equipment, etc.

[0031] Optionally, the above-mentioned multiple materials can be collected by one material collection device or by multiple material collection devices, and this application does not limit this.

[0032] As an example and not a limitation, the above-mentioned multiple materials are collected by three camera devices. These three camera devices can be used to simultaneously shoot the same scene from different angles and positions to obtain video materials respectively captured by the three camera devices. The three camera devices send their respective captured video materials to the electronic device. Alternatively, the above-mentioned multiple materials are collected by one camera device and two recording devices. The camera device shoots a scene containing actor A and actor B. One recording device is used to record actor A's voice, and the other recording device is used to record actor B's voice. The camera device sends the video materials it has collected to the electronic device, one recording device sends the audio materials it has collected of actor A to the electronic device, and the other recording device sends the audio materials it has collected of actor B to the electronic device.

[0033] After acquiring multiple materials, for any one of the multiple materials, you can use the following method to determine whether the material contains a time code, and extract the time code of the material if it contains a time code. The determination method is as follows:

[0034] For an i-th material among the multiple materials, detecting whether the i-th material includes a time code stream, where the i-th material is any material among the multiple materials, and i is a positive integer less than or equal to the number of the multiple materials;

[0035] If the i-th material contains a time code stream, the time code of the i-th material is extracted from the time code stream;

[0036] If the i-th material does not contain a time code stream, then check whether the i-th material contains time code metadata;

[0037] If the i-th material contains time code metadata, then extract the time code of the i-th material from the time code metadata;

[0038] If the i-th material does not include a time code stream and time code metadata, it is determined that the i-th material does not include a time code.

[0039] The time code stream can be a data sequence used to record and identify time information. The data in the material (taking the i-th material as an example) is written frame by frame. If it contains a time code stream, the time code of the data in the i-th material is also written frame by frame (that is, the time code is stored in the form of a time code stream). Time code parsing starts from the first frame data. If the time code is parsed before the N-th (N is an integer greater than 0, for example, N is 2) frame data is parsed, it can be determined that the i-th material contains a time code stream, and the parsed time code is determined as the time code of the i-th material. If the time code is still not parsed when the N-th frame data is parsed, it can be determined that the i-th material does not contain a time code stream.

[0040] Timecode metadata can refer to metadata that contains timecodes. For example, the timecode for the i-th clip can be written into a piece of metadata for the i-th clip. Since the i-th clip is typically stored as a file, timecode metadata can also be referred to as file timecode metadata, meaning the metadata that contains the timecode in the file corresponding to the i-th clip.

[0041] Because timecode is written frame by frame, extraction requires frame-by-frame analysis. Therefore, extracting the timecode for the i-th clip from the timecode stream can be more accurately performed. Therefore, you can first check whether the i-th clip contains a timecode stream. If it does not, extract the timecode for the i-th clip based on the timecode metadata.

[0042] If the i-th material does not contain the time code stream and the time code metadata, it can be determined that the i-th material does not contain the time code, thereby achieving automatic identification of the material that does not contain the time code.

[0043] Optionally, the identification of the material with incorrect time code among multiple materials can be performed manually or through other methods, which is not limited in this application.

[0044] Step 102 : When a first material exists among the multiple materials and the first material includes audio data, if a second material exists among the multiple materials, determine the time code of the first material based on the time code of the second material.

[0045] The first material is material that does not contain a time code or has an incorrect time code. As an example and not a limitation, the first material can be material generated during acquisition that does not contain a time code or has an incorrect time code. This application can effectively solve the problem of missing or inaccurate time codes in materials, thereby ensuring smooth post-production.

[0046] The second material is a material that includes audio data and time code, and the audio waveform of the included audio data matches the audio waveform of the audio data of the first material. It should be understood that if the first material contains an incorrect time code, then in order to distinguish it from the incorrect time code contained in the first material, the time code contained in the second material or the third material described below can be understood as the correct time code.

[0047] An audio waveform may refer to the representation of audio data in a time and amplitude coordinate system.

[0048] It should be understood that the number of first materials in the multiple materials may be zero, one, or at least two. When the number of first materials in the multiple first materials is zero, it indicates that the first material does not exist in the multiple materials. When the number of first materials in the multiple first materials is one, the time code of the first material can be obtained using the embodiment of the present application. When the number of first materials in the multiple first materials is at least two, the time code of each first material can be obtained using the embodiment of the present application.

[0049] In one embodiment, if the second material does not exist in the multiple materials, the first material may be marked as an abnormal material, and a corresponding abnormal reminder may be issued to remind the user that the first material is abnormal.

[0050] In one embodiment, when a first material exists among multiple materials and the first material includes audio data, the electronic device can first determine whether there is any remaining material among the multiple materials, where the remaining material is the material other than the first material among the multiple materials. If there is at least one remaining material, it is then determined whether there is a second material in the at least one remaining material. Based on this, it can be determined whether the second material exists among the multiple materials (i.e., the second material is searched for from the multiple materials).

[0051] In one possible implementation, the electronic device may first determine whether there is at least one material containing audio data in at least one remaining material; if there is at least one material containing audio data, then match the audio waveform of the audio data of each material in the at least one material containing audio data with the audio waveform of the audio data of the first material; if there is no material that matches the audio waveform of the audio data of the first material, then determine that there is no second material in the at least one remaining material, that is, there is no second material in the multiple materials; if there is material that matches the audio waveform of the audio data of the first material, then determine whether the material contains a time code; if the material contains a time code, then determine that the material is the second material; if the material does not contain a time code, then determine that the material is not the second material.

[0052] In another possible implementation, the electronic device may first determine whether there is at least one material containing a time code in at least one remaining material; if there is at least one material containing a time code, then determine whether there is at least one material containing audio data in at least one material containing a time code; if there is at least one material containing audio data, then match the audio waveform of the audio data of each material in the at least one material containing audio data with the audio waveform of the audio data of the first material; if there is no material that matches the audio waveform of the audio data of the first material, then determine that there is no second material in the at least one remaining material, that is, there is no second material in the multiple materials; if there is material that matches the audio waveform of the audio data of the first material, then determine that the material is the second material.

[0053] Since the time codes of different materials collected by the same material collection device do not overlap, there is no need to perform waveform matching between the first material and other materials collected by the same material collection device when searching for the second material. Figure 2 In steps 201 to 203 , a second material is searched from multiple materials.

[0054] Step 201: Group multiple materials according to identification information of a material acquisition device contained in each material.

[0055] The materials in the same material group include identification information of the material acquisition device, that is, the materials in the same material group are acquired by the same material acquisition device.

[0056] The identification information of the material collection device can be used to uniquely identify the material collection device. It should be understood that the identification information of the material collection device can be composed of one or more combinations of letters, numbers, Chinese characters, etc., and this application does not limit this.

[0057] Based on the identification information of the material acquisition device contained in each material, the electronic device can assign the materials acquired by the same material device to the same material group to ensure that the time codes of the materials in the same material group do not overlap. The materials acquired by different material acquisition devices can be assigned to different material groups. The time codes of the materials in different material groups may overlap. On this basis, the time code of the first material can be determined by waveform matching of different material groups.

[0058] Step 202 : When multiple material groups are obtained, if a candidate material group exists in at least one other material group, then the audio waveform of the audio data of the first material is matched with the audio waveform of the audio data of each third material in the candidate material group.

[0059] Among them, the third material is the material containing the second audio data and the time code, the candidate material group is the material group containing the third material in the other material groups, the other material groups are the material groups in multiple material groups except the target material group, and the target material group is the material group where the first material is located.

[0060] Step 203: If there is a material in each third material whose audio waveform matches the audio waveform of the audio data of the first material, then the material is determined as the second material.

[0061] The electronic device can search for a second material matching the first material from the candidate material group by matching the audio waveform of the audio data of the first material with the audio waveform of each third material in the candidate material group. If any third material in the candidate material group has a material whose audio waveform matches the audio waveform of the audio data of the first material, it can be determined that the material is the second material, i.e., the second material has been found from the candidate material group. If no third material in the candidate material group has a material whose audio waveform matches the audio waveform of the audio data of the first material, it is determined that the second material has not been found from the candidate material group.

[0062] To improve the efficiency of searching for the second material, in the three aforementioned implementations of searching for the second material, the electronic device can stop searching for the second material after finding one second material, thereby ensuring that the number of found second materials is one. By way of example and not limitation, when there are multiple candidate material groups, a second material matching the first material can be randomly searched from one of the candidate material groups. If the second material is not found in this candidate material group, another unmatched candidate material group is randomly selected and the second material is searched from this candidate material group, and so on, until the second material is found or all candidate material groups are exhausted. Of course, it is understood that matching priorities can also be specified for multiple candidate materials, or matching priorities for multiple candidate material groups can be pre-set according to some preset rules. The second material is first searched from the candidate material group with the highest priority. If the second material is not found, the second material is then searched from the candidate material group with the next highest priority until the second material is found or all candidate material groups are exhausted. If the second material is not found after traversing all candidate material groups, the first material can be marked as abnormal and a corresponding abnormality alert can be issued to inform the user that the first material is abnormal. Optionally, this application does not limit the specific content of the above preset rules.

[0063] Of course, it is understandable that the electronic device may continue searching for the second material after finding one, until all remaining materials or candidate material groups have been traversed. In this case, if there are at least two second materials, one material may be specified from the at least two second materials, and the time code of the first material may be determined based on the time code of the specified material. Alternatively, the material that best matches the first material may be selected from the at least two second materials, and the time code of the first material may be determined based on the time code of the material that best matches the first material. If similarity as described below is used to determine whether waveforms between materials match, the second material with the greatest similarity may be determined as the material that best matches the first material.

[0064] In one embodiment, the materials in multiple material groups are arranged in order of collection time from early to late, and the electronic device can match the audio waveform of the audio data of the first material with the audio waveform of the audio data of each third material in any candidate material group in sequence until the second material is found or all candidate material groups are traversed.

[0065] The acquisition time of each third material in the candidate material group reflects the order of acquisition of each third material. Starting from the first third material in the candidate material group, the audio waveform of the audio data of the first material is matched with the audio waveform of the audio data of each third material in turn. The order of acquisition time can be used to gradually narrow the waveform matching range, thereby improving the waveform matching efficiency.

[0066] As an example but not a limitation, the target material group includes material A1, material A2 and material A3 in sequence. Material A1 and material A3 do not contain a time code, and material A2 contains a time code. There are two candidate material groups, namely material group B and material group C. Material group B includes material B1, material B2 and material B3 in sequence. Material B1, material B2 and material B3 all contain a time code. Material group C includes material C1 and material C2 in sequence, both of which contain a time code. Taking material A1 as an example, material A1 can be first waveform matched with the materials in material group B. In the waveform matching process, material A1 is first waveform matched with material B1 (that is, the audio waveform of the audio data of material A1 is matched with the audio waveform of the audio data of material B1); if the audio waveform of the audio data of material A1 matches the audio waveform of the audio data of material B1, then material B1 is determined to be the second material; if the audio waveform of the audio data of material A1 does not match the audio waveform of the audio data of material B1, then material A1 is waveform matched with material B2; if If the audio waveform of the audio data of material A1 matches the audio waveform of the audio data of material B2, material B2 is determined to be the second material. If the audio waveform of the audio data of material A1 does not match the audio waveform of the audio data of material B2, material A1 is waveform-matched with material B3. If the audio waveform of the audio data of material A1 matches the audio waveform of the audio data of material B3, material B3 is determined to be the second material. If the audio waveform of the audio data of material A1 does not match the audio waveform of the audio data of material B3, it is determined that the second material has not been found in material group B. Based on the same waveform matching method, the second material is searched in material group C until the second material is found or material group C is exhausted.

[0067] In a possible implementation, materials in multiple material groups are arranged in order of acquisition time from earliest to latest, and when a time code of the material is a start time code and the first material is not the first material in the target material group, matching an audio waveform of audio data of the first material with an audio waveform of audio data of any third material includes:

[0068] If the end time code of the third material is later than the end time code of the material preceding the first material, the audio waveform of the audio data of the first material is matched with the audio waveform of the audio data of the third material.

[0069] The first material in the target material group may refer to the material that is collected earliest in the target material group. As an example and not a limitation, material A1 in the target material group is the first material, while materials A2 and A3 are not the first materials.

[0070] For any material, the end time code of the material can be determined based on the start time code of the material and the duration of the material.

[0071] If the end time of the third material is later than the end time code of the previous material of the first material, it means that the third material has the possibility of being time-code aligned with the first material, so the third material can be waveform-matched with the first material; if the end time of the third material is earlier or later than the end time code of the previous material of the first material, it means that the third material cannot be time-code aligned with the first material, so the third material can be directly excluded, that is, the third material will not be waveform-matched with the first material, thereby improving the efficiency of waveform matching.

[0072] Since the Mel Frequency Cepstrum Coefficient (MFCC) feature can effectively characterize the characteristics of audio data and reflect the change in the amplitude of audio data, it is possible to determine whether the audio waveforms of two audio data match based on the MFCC features of the two audio data. In one possible implementation, the matching method of the audio waveform of the audio data of the first material and the audio waveform of the audio data of any third material includes:

[0073] Extracting MFCC features of the audio data of the first material and MFCC features of the audio data of the third material respectively;

[0074] Calculating a similarity between the MFCC features of the audio data of the first material and the MFCC features of the audio data of the third material;

[0075] If the similarity exceeds the similarity threshold, determining that the audio waveform of the audio data of the first material matches the audio waveform of the audio data of the third material;

[0076] If the similarity does not exceed the similarity threshold, it is determined that the audio waveform of the audio data of the first material does not match the audio waveform of the audio data of the third material.

[0077] The MFCC feature extraction process may include pre-emphasis, framing, windowing, Fast Fourier Transform (FFT), power spectrum calculation, Mel filter bank, logarithm, Discrete Cosine Transform (DCT), and other steps.

[0078] Optionally, a similarity threshold can be set according to scenario requirements.

[0079] In one embodiment, before respectively extracting the MFCC features of the audio data of the first material and the MFCC features of the audio data of the third material, the method further includes:

[0080] The audio data of the first material and the audio data of the third material are resampled so that the sampling rates and data formats of the audio data of the first material and the audio data of the third material are the same.

[0081] The electronic device resamples the audio data of the first material and the audio data of the third material, and can adjust the sampling rate and data format of the audio data of the first material and the audio data of the third material, adjust the sampling rate of the audio data of the first material and the audio data of the third material to a preset sampling rate, thereby ensuring that the frequency components of the audio data of the first material and the audio data of the third material are consistent, avoiding spectrum aliasing differences, and adjust the data format of the audio data of the first material and the audio data of the third material to a preset data format, thereby ensuring that the data accuracy of the audio data of the first material and the audio data of the third material is consistent, and ensuring preprocessing consistency in the MFCC feature extraction process.

[0082] Optionally, the preset sampling rate and the preset data format can be set according to the scene requirements, which is not limited in this application. As an example and not a limitation, the preset sampling rate is 8000 Hz and the preset data format is 8-bit floating point.

[0083] In another embodiment, after obtaining multiple materials and determining the material containing audio data in the multiple materials, all the materials containing audio data in the multiple materials can be resampled so that the sampling rate and data format of all the materials containing audio data in the multiple materials are the same.

[0084] In one possible implementation, if a second pixel exists in multiple materials, the electronic device can determine the offset of the first material relative to the second material based on the waveform matching of the first material and the second material, and can determine the time code of the first material based on the time code of the second material and the offset of the first material relative to the second material. In particular, when determining whether the audio waveform of the audio data of the first material matches the audio waveform of the audio data of the third material by the similarity of the MFCC features of the audio data of the first material and the MFCC features of the audio data of the third material, the offset of the first material relative to the second material can be determined using the similarity of the MFCC features of the audio data of the first material and the MFCC features of the audio data of the second material, for example, different similarities correspond to different offsets. In particular, the offset of the first material relative to the second material can refer to the offset of the time code of the first material relative to the time code of the second material.

[0085] After the electronic device determines the time code of all the first materials among multiple materials, it can realize one-click synchronization of multiple materials on this basis, reducing the cost of manual intervention, improving the efficiency and accuracy of material synchronization, and providing time codes for materials without time codes or with incorrect time codes, thereby increasing the number of available materials and improving the availability of materials.

[0086] In one embodiment, when the time code of the material is a start time code, after determining the time code of the first material based on the time code of the second material, the method further includes:

[0087] For any first material, if the time period between the start time code and the end time code of the first material overlaps with the time period between the start time code and the end time code of the material containing the time code in the target material group, the first material is marked as a conflicting material;

[0088] And / or, when there are multiple first materials in the target material group, if there are at least two first materials whose time periods between the start time code and the end time code overlap, the at least two first materials are marked as conflicting materials.

[0089] The overlap of time periods of different materials includes partial overlap and complete overlap.

[0090] Because different materials in the same material group are collected by the same material acquisition device at different times, that is, the time periods between the start time codes and end time codes of different materials in the same material group do not overlap. Therefore, by determining whether the time periods between the start time code and the end time code of the first material and the material containing the time code in the target material group (the time code of this material is extracted from the time code stream or time code metadata, not determined based on waveform matching) overlap, and / or determining whether the time periods between the start time codes and end time codes of different first materials overlap, it is possible to further determine whether the first material is a conflicting material.

[0091] In an embodiment of the present application, after obtaining multiple materials, if there is a first material among the multiple materials and the first material contains audio data, it can be determined whether there is a second material among the multiple materials. The second material is a material that contains audio data and time code and the audio waveform of the audio data contained therein matches the audio waveform of the audio data of the first material. Since the audio data of the two materials with waveform matching are the same or relatively similar, the relationship between the time codes of the two materials can be determined based on the waveform matching. Therefore, if there is a second material among the multiple materials, the time code of the first material can be determined based on the time code of the second material, that is, the time code of the material that does not contain a time code or has an incorrect time code can be obtained based on the time code of the second material, thereby improving the availability of the material that does not contain a time code or has an incorrect time code.

[0092] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0093] Corresponding to the time code acquisition method described in the above embodiment, Figure 3 A schematic structural diagram of a time code acquisition device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0094] Reference Figure 3 , the device comprises:

[0095] The material acquisition module 301 is used to acquire multiple materials;

[0096] a time code determining module 302 for determining, when a first material exists among the plurality of materials and the first material includes audio data, a time code of the first material based on a time code of the second material if a second material exists among the plurality of materials;

[0097] The first material is a material that does not contain a time code or has an incorrect time code, and the second material is a material that contains audio data and a time code, and the audio waveform of the audio data contained therein matches the audio waveform of the audio data of the first material.

[0098] Optionally, the time code determination module 302 includes:

[0099] a material grouping unit, configured to group the plurality of materials according to identification information of a material acquisition device contained in each material, wherein materials in the same material group contain the same identification information of the material acquisition device;

[0100] a waveform matching unit configured to, when multiple material groups are obtained, match the audio waveform of the audio data of the first material with the audio waveform of the audio data of each third material in the candidate material group if a candidate material group exists in at least one other material group, wherein the third material is a material including audio data and a time code, the candidate material group is a material group including the third material in the other material groups, the other material groups are material groups other than a target material group in the multiple material groups, and the target material group is the material group including the first material;

[0101] The material determining unit is configured to determine, if there is a material in each of the third materials whose audio data has an audio waveform matching the audio waveform of the audio data of the first material, the material as the second material.

[0102] Optionally, the materials in the multiple material groups are arranged in order of acquisition time from earliest to latest. When the time code of the material is a start time code and the first material is not the first material in the target material group, the waveform matching unit is specifically configured to:

[0103] If the end time code of the third material is later than the end time code of the previous material of the first material, the audio waveform of the first audio data is matched with the audio waveform of the second audio data of the third material.

[0104] Optionally, the waveform matching unit is specifically used to:

[0105] extracting MFCC features of the audio data of the first material and MFCC features of the audio data of the third material respectively;

[0106] Calculating the similarity between the MFCC features of the audio data of the first material and the MFCC features of the audio data of the third material;

[0107] If the similarity exceeds a similarity threshold, determining that the audio waveform of the audio data of the first material matches the audio waveform of the audio data of the third material;

[0108] If the similarity does not exceed the similarity threshold, it is determined that the audio waveform of the audio data of the first material does not match the audio waveform of the audio data of the third material.

[0109] Optionally, the time code determination module 302 further includes:

[0110] A resampling unit is configured to resample the audio data of the first material and the audio data of the third material so that the sampling rates and data formats of the audio data of the first material and the audio data of the third material are the same.

[0111] Optionally, when the time code of the material is a start time code, the apparatus further includes a marking module, the marking module being configured to:

[0112] For any of the first materials, if a time period between a start time code and an end time code of the first material overlaps with a time period between a start time code and an end time code of a material containing a time code in the target material group, marking the first material as a conflicting material;

[0113] And / or, in the case where there are multiple first materials in the target material group, if there are at least two first materials among the multiple first materials whose time periods between the start time code and the end time code overlap, the at least two first materials are marked as conflicting materials.

[0114] Optionally, the above device further includes:

[0115] a code stream detection module, configured to detect, for an i-th material among the plurality of materials, whether the i-th material includes a time code stream, where the i-th material is any material among the plurality of materials, and i is a positive integer less than or equal to the number of the plurality of materials;

[0116] a first extraction module, configured to extract the time code of the i-th material from the time code stream if the i-th material includes the time code stream;

[0117] a metadata detection module, configured to detect whether the second material contains time code metadata if the i-th material does not contain the time code stream;

[0118] a second extraction module, configured to extract the time code of the i-th material from the time code metadata if the i-th material includes the time code metadata;

[0119] The determining module is configured to determine that the i-th material does not include the time code if the i-th material does not include the time code stream and the time code metadata.

[0120] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0121] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4 Only one is shown), a memory 41 and a computer program 42 stored in the memory 41 and executable on the at least one processor 40, wherein the processor 40 implements the steps of any of the above-mentioned method embodiments when executing the computer program 42.

[0122] The electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that Figure 4 This is merely an example of the electronic device 4 and does not constitute a limitation on the electronic device 4 . The electronic device 4 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 4 may also include input and output devices, network access devices, etc.

[0123] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0124] In some embodiments, the memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. In other embodiments, the memory 41 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 4. Furthermore, the memory 41 may also include both an internal storage unit of the electronic device 4 and an external storage device. The memory 41 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0127] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0128] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0131] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A time code acquisition method, characterized in that: include: Get multiple materials; If a first material exists among the plurality of materials and the first material includes audio data, if a second material exists among the plurality of materials, determining the time code of the first material based on the time code of the second material; The first material is a material that does not contain a time code or has an incorrect time code, and the second material is a material that contains audio data and a time code, and the audio waveform of the audio data contained therein matches the audio waveform of the audio data of the first material.

2. The time code acquisition method according to claim 1, characterized in that: The method for determining the second material includes: grouping the multiple materials according to identification information of a material acquisition device contained in each of the multiple materials, wherein the materials in the same material group contain the same identification information of the material acquisition device; In the case where multiple material groups are obtained, if a candidate material group exists in at least one other material group, matching the audio waveform of the audio data of the first material with the audio waveform of the audio data of each third material in the candidate material group, where the third material is a material including audio data and a time code, the candidate material group is a material group including the third material, the other material groups are material groups other than the target material group in the multiple material groups, and the target material group is the material group including the first material; If there is a material in each of the third materials whose audio waveform of the audio data matches the audio waveform of the audio data of the first material, then the material is determined to be the second material.

3. The time code acquisition method according to claim 2, characterized in that: The materials in the plurality of material groups are arranged in order of acquisition time from earliest to latest, and when a time code of the material is a start time code and the first material is not the first material in the target material group, matching an audio waveform of audio data of the first material with an audio waveform of audio data of any of the third materials, including: If the end time code of the third material is later than the end time code of the previous material of the first material, the audio waveform of the audio data of the first material is matched with the audio waveform of the audio data of the third material.

4. The time code acquisition method according to claim 2 or 3, characterized in that: The matching method of the audio waveform of the audio data of the first material and the audio waveform of the audio data of any of the third materials includes: extracting MFCC features of the audio data of the first material and MFCC features of the audio data of the third material respectively; Calculating the similarity between the MFCC features of the audio data of the first material and the MFCC features of the audio data of the third material; If the similarity exceeds a similarity threshold, determining that the audio waveform of the audio data of the first material matches the audio waveform of the audio data of the third material; If the similarity does not exceed the similarity threshold, it is determined that the audio waveform of the audio data of the first material does not match the audio waveform of the audio data of the third material.

5. The time code acquisition method according to claim 4, characterized in that: Before respectively extracting the MFCC features of the audio data of the first material and the MFCC features of the audio data of the third material, the method further includes: The audio data of the first material and the audio data of the third material are resampled so that the sampling rates and data formats of the audio data of the first material and the audio data of the third material are the same.

6. The time code acquisition method according to claim 2 or 3, characterized in that: In a case where the time code of the material is a starting time code, after determining the time code of the first material based on the time code of the second material, the method further includes: For any of the first materials, if a time period between a start time code and an end time code of the first material overlaps with a time period between a start time code and an end time code of a material containing a time code in the target material group, marking the first material as a conflicting material; And / or, in the case where there are multiple first materials in the target material group, if there are at least two first materials among the multiple first materials whose time periods between the start time code and the end time code overlap, the at least two first materials are marked as conflicting materials.

7. The time code acquisition method according to any one of claims 1 to 3, characterized in that: After acquiring multiple materials, including: For an i-th material among the multiple materials, detecting whether the i-th material includes a time code stream, where the i-th material is any material among the multiple materials, and i is a positive integer less than or equal to the number of the multiple materials; If the i-th material includes the time code stream, extracting the time code of the i-th material from the time code stream; If the i-th material does not include the time code stream, detecting whether the i-th material includes time code metadata; If the i-th material includes the time code metadata, extracting the time code of the i-th material from the time code metadata; If the i-th material does not include the time code stream and the time code metadata, it is determined that the i-th material does not include the time code.

8. A time code acquisition device, characterized in that: include: Material acquisition module, used to acquire multiple materials; a time code determining module configured to, when a first material exists among the plurality of materials and the first material includes audio data, determine a time code of the first material based on a time code of the second material if a second material exists among the plurality of materials; The first material is a material that does not contain a time code or has an incorrect time code, and the second material is a material that contains audio data and a time code, and the audio waveform of the audio data contained therein matches the audio waveform of the audio data of the first material.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the time code acquisition method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the time code acquisition method according to any one of claims 1 to 7 to be executed.

Images