Method, device and system for audio data processing and watermark extraction

Abstract

Disclosed are an audio data processing and watermark extraction method, device and system. The audio data processing method may include: generating a watermark code that characterizes the characteristics of the audio data; embedding a portion of the watermark code into a plurality of audio data as a digital watermark, wherein the digital watermarks of two or more of the audio data can be extracted to restore the watermark code. The watermarked audio data can be published by the platform and used by users for collaborative creation. The present invention uniformly extracts and encodes protection information for a number of shorter audio creation contents based on the characteristics of multiple uses and co-occurrences, and embeds the information into the position where the corresponding audio appears in a piece of music, thereby solving the problem that it is difficult to embed a longer digital watermark into a shorter audio, and at the same time improving the robustness of the watermark through a low bit rate.

Description

Technical Field

[0001] The present disclosure relates to a digital media processing technology, and in particular to a method, device and system for audio data processing and watermark extraction. Background Art

[0002] At present, copyright protection for music creation is often based on longer music clips or entire songs, with public release as the core evidence. Generally, the first public release enjoys the copyright. Shorter clips, beat designs, and more characteristic sound effects are difficult to release separately, so they are more difficult to protect separately. These music elements are important copyright data for collaborative creation platforms that assist users in creation, as well as uploading users.

[0003] In the prior art, digital watermarks are usually added to audio for copyright protection. However, this is usually used by the publisher of the work to protect the copyright of a complete work. In the scenario of collaborative creation, since one party's creation may be fragmented and not actively involved, it is difficult to use existing watermarks for protection, especially for the protection of shorter audio clips.

[0004] To this end, a technical solution is needed that can protect the copyright of short audio. Summary of the invention

[0005] A technical problem to be solved by the present disclosure is to provide an audio data processing solution, which groups the protection information of shorter audio creation content into shorter source codes for digital watermark encoding, wherein each short audio includes a shorter digital watermark and a specific number of watermarks can restore the protection information, thereby being able to add a watermark to the short audio without affecting the quality of the short audio, and restoring the protection information for the short audio from the extracted watermark when other musical works frequently quote the short audio.

[0006] According to a first aspect of the present disclosure, there is provided an audio data processing method, comprising: generating a watermark code characterizing features of the audio data; and embedding a portion of the watermark code into a plurality of audio data as a digital watermark, wherein the digital watermarks of two or more of the audio data can be extracted for restoring the watermark code.

[0007] According to a second aspect of the present disclosure, there is provided a method for processing audio and video data, comprising: generating a watermark code characterizing features of the audio and video data; embedding a portion of the watermark code as a digital watermark into a plurality of audio and video data, each of which comprises audio data and image frame data, wherein the digital watermarks of two or more of the audio and video data can be extracted for restoring the watermark code.

[0008] According to a third aspect of the present disclosure, a music composition method is provided, comprising adding the audio data embedded with the digital watermark described in the first aspect to a music work.

[0009] According to a fourth aspect of the present disclosure, a digital watermark extraction method is provided, comprising: dividing an acquired musical work into a plurality of audio segments to be detected; determining whether the plurality of audio segments to be detected contain the audio data embedded with the digital watermark as described in the first aspect; and extracting the digital watermark from the audio segment to be detected containing the audio data.

[0010] According to a fifth aspect of the present disclosure, a method for extracting streaming media watermarks is provided, comprising: acquiring streaming media data; detecting segment by segment whether the streaming media data contains audio data embedded with the digital watermark as described in the first aspect; extracting the digital watermark in the segment containing the audio data; and in the case of extracting multiple different digital watermarks embedded in two or more of the audio data, restoring the watermark code from the multiple different digital watermarks.

[0011] According to a sixth aspect of the present disclosure, a method for adding a streaming media watermark is provided, comprising adding audio data or image frame data as described in the first or second aspect to generate a streaming media work embedded with the digital watermark.

[0012] According to the seventh aspect of the present disclosure, there is provided an audio data processing device, comprising: a coding generation unit, for generating a watermark code characterizing characteristics of audio data; a watermark embedding unit, for embedding a part of the watermark code into a plurality of audio data as a digital watermark, wherein the digital watermarks of two or more of the audio data can be extracted for restoring the watermark code.

[0013] According to an eighth aspect of the present disclosure, a digital watermark extraction device is provided, comprising: a segmentation unit, used to divide an acquired musical work into a plurality of audio segments to be detected; a matching unit, used to determine whether the plurality of audio segments to be detected contain the audio data embedded with the digital watermark as described in the first aspect; and an extraction unit, used to extract the digital watermark from the audio segment to be detected containing the audio data.

[0014] According to a ninth aspect of the present disclosure, a music collaborative creation system is provided, comprising a server and multiple clients, wherein the server is used to: publish multiple audio data embedded with digital watermarks as described in the first aspect, and the client is used to: use one or more of the multiple audio data for music collaborative creation; and publish the created music works.

[0015] According to a tenth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the methods described in the first to sixth aspects above.

[0016] According to an eleventh aspect of the present disclosure, a non-temporary machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the methods described in the first to sixth aspects above.

[0017] Therefore, we can extract and encode the protection information of several shorter audio creations in a unified way according to the characteristics of high frequency use and common appearance, and embed them into the position where the corresponding audio appears in a piece of music. This can solve the problem that it is difficult to embed a longer digital watermark into a shorter audio, improve the robustness of the watermark technology through low bit rate, and reduce the impact of watermark embedding on the sound quality of each short audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0019] Figure 1 A schematic flow chart of an audio data processing method according to an embodiment of the present invention is shown.

[0020] Figure 2 A schematic flow chart of a digital watermark extraction method according to an embodiment of the present invention is shown.

[0021] Figure 3 A schematic diagram showing the composition of an audio data processing device according to an embodiment of the present invention is shown.

[0022] Figure 4 A schematic diagram showing the composition of a digital watermark extraction device according to an embodiment of the present invention is shown.

[0023] Figure 5 A schematic diagram showing the composition of a music collaborative creation system according to an embodiment of the present invention is shown.

[0024] Figure 6 An example of copyright protection based on the audio data processing and watermark extraction operations of the present invention is shown.

[0025] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned audio data processing, collaborative creation and / or watermark extraction method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0027] At present, copyright protection for music creation is often based on longer music clips or entire songs, with public release as the core evidence. Generally, the first public release enjoys the copyright. Shorter clips, beat designs, and more characteristic sound effects are difficult to release separately, so they are more difficult to protect separately. These music elements are important copyright data for collaborative creation platforms that assist users in creation, as well as uploading users.

[0028] In the prior art, digital watermarks are usually added to audio for copyright protection. However, this is usually used by the publisher of the work to protect the copyright of a complete work. In the scenario of collaborative creation, since one party's creation may be fragmented and not actively involved, it is difficult to use existing watermarks for protection, especially for the protection of shorter audio clips. The present invention needs to protect original music works, especially short music works, which are created by the author himself, are not plagiarized or imitated, have unique personality in both content and form, and have social consensus value.

[0029] To this end, the present disclosure provides an audio data processing solution, which groups the protection information of shorter audio creation content into shorter source codes for digital watermark encoding. Each short audio includes a shorter digital watermark and a specific number of watermarks can restore the protection information. In this way, a watermark can be added to the short audio without affecting the quality of the short audio, and when other musical works quote the above short audio in large quantities, the protection information for the above short audio can be restored from the extracted watermark.

[0030] Figure 1 A schematic flow chart of an audio data processing method according to an embodiment of the present invention is shown.

[0031] In step S110, a watermark code characterizing the characteristics of the audio data is generated. In step S120, a part of the watermark code is respectively embedded into a plurality of audio data as a digital watermark, wherein the digital watermarks of two or more of the audio data can be extracted to restore the watermark code.

[0032] Here, audio data refers to audio data that is usually shorter, such as audio clips, sound effect clips, melody clips, or beat clips of a few seconds. A chord refers to a group of sounds with a certain interval relationship, that is, three or more notes are combined vertically in a third or non-third degree superposition relationship, which is called a chord. Beat is a unit of rhythm measurement. In music, a series of beats with certain strengths and weaknesses are repeated at regular intervals. For example, 2 / 4, 4 / 4, 3 / 4 beats, etc. Specific chords or beats can produce special sound presentation effects.

[0033] These short audio data usually cannot be published as an independent work, but are usually added by the creator to his or her own works, such as audio works, karaoke works, video clips, short videos, etc.

[0034] Considering that these short audio data may be cited multiple times or appear together in a piece of music, the present invention uniformly extracts and encodes protection information for several shorter audio creation contents. These codes can be embedded in the position where the corresponding audio appears in a piece of music. This solves the problem that it is difficult to embed a longer digital watermark into a shorter audio, improves the robustness of the watermark technology through a low bit rate, and reduces the impact of watermark embedding on the sound quality of each short audio.

[0035] Here, the watermark encoding that characterizes the characteristics of the audio data can be generated based on the copyright protection information. To this end, the method can also include generating copyright protection information for one or a group of audio data (i.e., music creation content). The source of these contents can be produced by the platform, produced by, for example, AI technology, uploaded by users, and so on. The generated copyright protection information can be used to characterize the basic information and uniqueness of the piece or group of creative content, as a mark of originality tracing and suitable for public use. Such as the ID, author, creation time, name or other unique identifiers of the work.

[0036] Therefore, in step S110, in order to generate a watermark code, the copyright protection information generated as above can be encoded independently or in batches, and encryption or non-encryption can be selected. To this end, step S110 can include generating the watermark code representing the audio data characteristics of the one or a group of audio data based on the (one) copyright protection information. As an alternative, step S110 can also generate the watermark code representing the audio data characteristics of multiple or multiple groups of audio data based on multiple copyright protection information. Specifically, the watermark code representing the audio data characteristics of multiple or multiple groups of audio data can be generated by interleaving encoding.

[0037] Therefore, through batch encoding (for example, interleaved encoding) and encryption encoding, for example, the specific encoding and encryption method known only to the music collaborative creation platform that implements the encoding can effectively hide the content of the protected information and reduce the risk of digital watermarks being cracked.

[0038] For example, the music collaborative creation platform of the present invention uses AI to generate 50 sound effects for scene A (e.g., funny scene). The platform can organize data such as platform name, generation method, applicable scene, creation time, etc. to generate copyright protection information for the 50 sound effects. Subsequently, the platform can similarly use AI to generate sound effects for other scenes B, C, and D (e.g., dance scene, ancient style scene, hip-hop scene) (e.g., distributed as 50, 30, and 40 sound effects), and generate respective copyright protection information accordingly.

[0039] The copyright protection information of scenes B, C and D and the copyright protection information of scene A can be batch-encoded by the platform. Specifically, the above four pieces of copyright protection information can be converted into binary codes, combined together for interleaved coding, and then the encoded results are encrypted according to a certain agreed encryption algorithm. Thus, the watermark code that can represent the characteristics of the above audio data (four groups, a total of 170 sound effects) is obtained. The above watermark code is, for example, a 1048-bit binary code.

[0040] Subsequently, in step S120, a portion of the above-mentioned 1048-bit binary code can be embedded into each sound effect segment of the 170 sound effects, for example, a continuous 16-bit or 32-bit code selected based on a specific rule in the 1048-bit binary code, as a digital watermark of this sound effect segment, thereby obtaining 170 sound effects embedded with digital watermarks.

[0041] When a user creates a musical work, multiple sound effects are usually added. When a musical work is released or suspected of being plagiarized, the suspected work can be detected. When only one sound effect is detected, the digital watermark recovered therefrom, such as 16-bit or 32-bit encoding, usually cannot restore the above-mentioned copyright protection information. When two or more sound effects are detected (for example, when the 64-bit or 128-bit encoding can be recovered and combined), the digital watermarks recovered therefrom can be combined to restore the watermark encoding, and the copyright protection information can be restored based on the encoding and / or encryption means known to the platform.

[0042] Here, it should be understood that the present invention does not limit the specific encoding method, encryption method, and the way in which the generated watermark code is selected as the embedded digital watermark in segments, as long as the watermark data can be detected from the audio segment during subsequent detection, and multiple watermark data can be combined to restore meaningful data. For example, for watermark codes that do not have a fixed length and do not limit the number of audio segments, rateless coding can be preferably used to facilitate the recovery of meaningful copyright protection information from the unspecified digital watermark combination detected from the unspecified audio segment.

[0043] Furthermore, the length of the digital watermark contained in a single audio segment and the number of digital watermarks of the audio segment required to recover meaningful data therefrom can be determined based on, for example, copyright protection rules, the length of the audio segment, etc. In other words, two or more digital watermarks of the audio data can be extracted to recover the watermark code, which can include: determining the number of different digital watermarks required to recover the watermark code based on the rules. That is, in different situations, two, five, ten, or even more different digital watermarks may be required to restore the watermark code.

[0044] For example, when a user uses an audio clip with a digital watermark added according to the present invention only once in a whole work (e.g., 2 seconds long), the use of the audio clip should not be considered as infringement, and the digital watermark extracted from the audio clip does not contain sufficient effective data length to restore the copyright information. However, when a user uses an audio clip with a digital watermark added according to the present invention ten times in a whole work (e.g., located at different positions and a total of 25 seconds long), the use of the audio clip can be considered as infringement, and the digital watermarks extracted from the ten audio clips contain sufficient effective data length to restore the copyright information.

[0045] As mentioned above, the digital watermarks used to recover the watermark coding need to be different digital watermarks. To this end, when a user repeatedly adds the same short audio clip to a piece of music, the same audio clip embedded with different digital watermarks can be provided each time it is added. To this end, step S120 may include: embedding a part of the watermark coding as a digital watermark into multiple audio data copies of the one audio data. Specifically, for shorter platform creation content, its protection information is grouped into shorter source codes for digital watermark coding, and each time the audio is called, a different group of watermarks is called to be embedded in the audio. When the content is called multiple times in a piece of music, the different groups of coding ensure that there are enough code elements to complete accurate information decoding. For example, each time the user calls a sound effect (for example, sound effect AA), a sound effect AA with a different digital watermark is inserted. To this end, when information is subsequently extracted, these different digital watermarks from the same sound effect AA can provide sufficient data length to recover copyright information.

[0046] Although the audio data processing scheme of the present invention is mainly applicable to short audio, the above scheme of the present invention can also be applied to a complete musical work. To this end, step S110 may include generating a watermark code that characterizes the characteristics of a musical work; step S120 may include: embedding a part of the watermark code into a plurality of audio data constituting the musical work as a digital watermark. Thus, by embedding a digital watermark of a musical work in segments, and the digital watermarks of several audio segments can restore effective information, the ability to distinguish plagiarism of segments of the overall musical work can be improved.

[0047] In addition, when the same user uses the platform to create, since the platform can know all the user's behavioral operations, the different creative contents called by it can be uniformly source encoded, and the encoded code elements are embedded in the music along with different audio clips to ensure that there is enough information to decode and obtain protection information. To this end, the method can also include obtaining the audio data called by the creative entity, and step S110 can include generating a unified watermark code representing the called audio data, and step S120 can include embedding a part of the unified watermark code into the called audio clips in the music created by the creative entity as a digital watermark. In other words, in addition to pre-embedding short audio with watermarks, unified watermark code generation and subsequent embedding can also be performed for a certain creative entity (for example, a user) or one or a batch of works created by it.

[0048] For the copyright protection information and / or the watermark coding information generated thereby, you can choose to keep it confidential or publish it. Usually, in order to facilitate tracing, it is necessary to publish the above information on a relatively credible platform. To this end, the method may also include: publishing the watermark code and / or the copyright protection information used to generate the watermark code. The publishing may include: publishing to the outside through the platform; publishing the generation time of the watermark code and / or the copyright protection information; or publishing using blockchain. Therefore, when this information is decoded from a piece of music claiming to be original, it is proved through this information that the copyright of this music is wholly or partially owned by the platform.

[0049] In addition, multiple audio data can also be published through the music creation platform. The published audio data can be audio data embedded with a digital watermark, so when the user uses the above audio data to generate a music work and publishes it, the published music work contains the above digital watermark. In addition, the published audio data can also be audio data without a digital watermark embedded, so when the user uses the above audio data to generate a music work and publishes it, it can be uniformly encoded and embedded based on the audio data contained in the completed music work, and the music work published for this purpose also contains the above digital watermark.

[0050] In addition, these audio data can be input into the retrieval library to facilitate subsequent audio positioning and watermark extraction.

[0051] The audio data as described above may be data included in a pure audio file or media, or data included in a video file or media. To this end, the present invention may also be implemented as an audio and video data processing method, comprising: generating a watermark code that characterizes the characteristics of the audio and video data; embedding a portion of the watermark code as a digital watermark into a plurality of audio and video data each including audio data and image frame data, wherein the digital watermarks of two or more of the audio and video data may be extracted to restore the watermark code.

[0052] Here, the watermark embedding for audio and video data can be the separate embedding of video data and audio data, or the associated watermark can be generated for video and audio data and embedded, or the associated embedding can be performed. For this purpose, embedding a part of the watermark code into multiple audio and video data as a digital watermark can include: embedding a part of the watermark code into the audio data and image frame data included in the audio and video data as a digital watermark; or associatively embedding a part of the watermark code into the audio data and image frame data included in the audio and video data as a digital watermark.

[0053] According to another aspect of the present invention, it can also be implemented as a music creation method. The method can be executed by a user, for example, by a user using a computing device, such as a mobile phone, on which the above-mentioned platform APP is installed. Users can select short audios as described above in the APP for collaborative music creation. These short audios can be created by other users or entities on the platform. Here, collaborative creation refers to the process of jointly creating a work by two or more people. Collaborative creation concentrates the wisdom of many people, which can reduce the time of music creation and improve the quality of the work.

[0054] Specifically, users can add the above-mentioned original audio data to their own music works. Subsequently, users can also use the platform to publish their created music works. As mentioned above, users can directly add audio data embedded with digital watermarks, or after the music work is completed, the platform can uniformly encode and embed watermarks on the audio data added by the user according to the user's operation.

[0055] It should be understood that, here, the musical work can be a pure audio work or an audio-visual work including images, as long as audio data processed as described in the present invention can be added to these works.

[0056] After the audio data processed according to the present invention or the music work containing the above audio data is released, it may be used by other platforms or users for secondary creation, etc. When there is doubt about the copyright of the secondary creation, the digital watermark extraction method of the present invention can be executed.

[0057] Figure 2 A schematic flow chart of a digital watermark extraction method according to an embodiment of the present invention is shown. To this end, in step S210, the acquired music piece can be divided into multiple audio segments to be detected. In step S220, it is determined whether the multiple audio segments to be detected contain the audio data embedded with the digital watermark as described above. In step S230, the digital watermark in the audio segment to be detected containing the audio data is extracted.

[0058] Furthermore, in the case where two or more different digital watermarks embedded in the audio data are extracted, the watermark code can be restored from the multiple different digital watermarks.

[0059] Subsequently, the copyright protection information used to generate the watermark code can be restored from the watermark code. The restored copyright protection information can be compared with the information previously released by the platform, and the originality of the audio content can be declared accordingly.

[0060] As mentioned above, the audio data that needs to be protected and contains a digital watermark when published by the user can be input into the retrieval library to facilitate the subsequent audio positioning and watermark extraction. To this end, in the watermark extraction stage, step S220 can include searching the retrieval library including the audio data; and determining whether the multiple audio segments to be detected contain the audio data embedded with the digital watermark according to the retrieval hit result.

[0061] According to the present invention Figure 1 The music work with digital watermark generated by the embodiment can be directly (or after being tampered by a third party) published on a streaming media website (for example, an audio or video website). After the music work is published on the audio or video website, it can be obtained by web page visitors in the form of streaming media. Here, streaming media refers to a technology and process of sending a series of multimedia data (usually compressed data) through the Internet in segments, and transmitting audio and video on the Internet in real time for viewing. Streaming media data can refer to multimedia data sent in segments through the Internet. This technology allows data packets to be sent like running water and continuously listened to and / or watched without downloading the entire media file before use.

[0062] When extracting watermarks, watermark extraction can be performed on a complete file, or on an audio stream or an audio / video stream in the form of streaming media. To this end, the present invention can also be implemented as a streaming media watermark extraction method, and includes: obtaining streaming media data; detecting whether the streaming media data contains the audio data embedded with the digital watermark as described above; extracting the digital watermark in the segment containing the audio data; and in the case of extracting multiple different digital watermarks embedded in two or more of the audio data, restoring the watermark code from the multiple different digital watermarks.

[0063] Here, streaming media may include: published videos or short videos; live content; and cloud interactive content. Specifically, the audio watermark retrieval according to the present invention or the joint watermark retrieval of audio and video can be performed for the published short videos. In addition, for online content, such as online live broadcasts, cloud interactions (including cloud pk with multi-party participation, etc.), watermark extraction operations can be performed based on the acquisition of streaming media.

[0064] Similarly, the present invention can also be implemented as a method for adding a watermark to a streaming media, including adding the audio data or image frame data as described above to generate a streaming media work embedded with the digital watermark. Similarly, the streaming media can include: published videos or short videos; live content; and cloud interactive content.

[0065] In addition, the present invention can also be implemented as an audio data processing device. Figure 3A schematic diagram showing the composition of an audio data processing device according to an embodiment of the present invention is shown.

[0066] As shown in the figure, the apparatus 300 includes a code generation unit 310 and a watermark embedding unit 320 .

[0067] The code generation unit 310 may be configured to generate a watermark code that represents characteristics of the audio data.

[0068] The watermark embedding unit 320 can be used to embed a portion of the watermark code into multiple audio data as a digital watermark. Two or more digital watermarks of the audio data (for example, different digital watermarks above a set threshold) can be extracted to restore the watermark code.

[0069] Accordingly, the present invention can also be implemented as a digital watermark extraction device. Figure 4 A schematic diagram showing the composition of a digital watermark extraction device according to an embodiment of the present invention is shown.

[0070] As shown in the figure, the apparatus 400 includes a segmentation unit 410 , a matching unit 420 , and an extraction unit 430 .

[0071] The segmentation unit 410 is used to divide the acquired music piece into a plurality of audio segments to be detected.

[0072] The matching unit 420 is used to determine whether the multiple audio segments to be detected include the audio data embedded with the digital watermark as described above.

[0073] The extraction unit 430 is used to extract the digital watermark from the audio segment to be detected that contains the audio data.

[0074] Furthermore, the apparatus 400 may also include a synthesis unit 440 for restoring the watermark code from the multiple different digital watermarks when multiple different digital watermarks embedded in two or more audio data are extracted.

[0075] As mentioned above, the audio data of the present invention can be watermarked and published by a music collaborative creation platform, and users can use these short audios for collaborative creation and publication.

[0076] To this end, the present invention can also be implemented as a music collaborative creation system, including a server and multiple clients. Figure 5 A schematic diagram showing the composition of a music collaborative creation system according to an embodiment of the present invention is shown.

[0077] As shown, system 500 may include a server 510 and multiple clients 520 .

[0078] The server 510 is used to provide a plurality of audio data processed by the method described above.

[0079] The client 520 is used to: use one or more of the multiple audio data to perform collaborative music creation; and publish the created music work, where the music work includes one or more of the digital watermarks.

[0080] Furthermore, the short audio with a watermark may also be created by a platform user, and the user who creates the short audio and the user who uses the short audio to create a musical work may be (and usually is) a different user.

[0081] To this end, the client 520 is also used to create and publish audio data. Correspondingly, the server 510 is used to add a digital watermark to the audio data created and published by the client using the processing method described above.

[0082] In addition, server 510 is also used to: execute the digital watermark extraction method as described above for the acquired music works. Existing music collaborative creation platforms include a large number of self-produced and user-uploaded sound effects, and will subsequently produce a large number of music clips, beats, chord combinations and other data to assist users in music creation. Therefore, for works that use a certain proportion of such data, the copyright can be at least partially regarded as belonging to the data creation user or platform. Through the audio processing and watermark extraction technical means of the present invention, the platform original content that may be used in each music work can be discovered, and copyright protection can be initiated when necessary.

[0083] Figure 6 An example of copyright protection based on the audio data processing and watermark extraction operations of the present invention is shown.

[0084] As shown in stage I, the collaborative creation platform (the "cloud" in the figure) can obtain the audio content (e.g., short audio, including sound effects, chords, beats, etc.) created and uploaded by the user, and perform the Figure 1 The processing shown. Figure 6 Although the above short audio is represented by a single note, it should be understood that the above representation is only used to illustrate the relative length of the audio, and does not imply that the short audio contains only one note.

[0085] Specifically, for each piece or group of original short audio content such as sound effects and music clips that can be used for music creation, information for copyright protection is generated. The source of these contents can be produced by the platform, produced through technology, uploaded by users, etc. The generated information is used to characterize the basic information and uniqueness of the piece or group of creative content, as a mark of originality tracing and suitable for public disclosure, such as the ID, author, creation time, name or other unique identifiers of the work.

[0086] The copyright protection information generated as above is encoded in batches or independently, and encryption or non-encryption is selected. For example, 100 pieces of the above information can be converted into binary codes, combined together for interleaved coding, and then the encoded results are encrypted according to a certain encryption algorithm.

[0087] The above copyright information or encoded information can be kept confidential, announced on the platform, or published on the blockchain. When this information is decoded from a piece of music claiming to be original, it can be used to prove that the copyright of the music is wholly or partially owned by the platform.

[0088] The platform can then embed this information into the corresponding piece or group of creative content in the form of a digital watermark.

[0089] The above operation can be illustrated by the operation of "adding a stamp to a single note" in the cloud of Phase I.

[0090] As shown in the figure, the created content can be input into the search library together with the watermark. The cylindrical object on the left side of stage I can represent not only the search library but also the database storing short audio.

[0091] Then, in Phase II, the platform can obtain these short audios from the database and publish these watermarked contents to users.

[0092] Users can upload music works including the above audio to the server and publish them.

[0093] In Phase III, the platform can obtain content with copyright issues from the Internet, segment it using audio processing technology, search each segment in the retrieval library, and extract the corresponding stored protection information when there is a hit.

[0094] In Phase IV, the platform can extract digital watermarks from music clips that match the protection information and compare the results with the protection information. When the credibility and matching degree are high, the platform is deemed to have copyright rights to the music and can issue a warning of suspected infringement to the above content.

[0095] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned audio data processing, collaborative creation and / or watermark extraction method according to an embodiment of the present invention is shown.

[0096] See also Figure 7 , the computing device 700 includes a memory 710 and a processor 720 .

[0097] The processor 720 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 720 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0098] The memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 720 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 710 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0099] The memory 710 stores executable codes, which, when processed by the processor 720, can enable the processor 720 to execute the audio data processing, collaborative creation and / or watermark extraction methods mentioned above.

[0100] The audio data processing scheme according to the present invention has been described in detail above with reference to the accompanying drawings. This scheme can extract and encode protection information for a number of shorter audio creations in a unified manner based on the characteristics of high frequency use and common appearance, and embed the information into the location where the corresponding audio appears in a piece of music. This solves the problem that it is difficult to embed a longer digital watermark into a shorter audio, improves the robustness of the watermark technology through a low bit rate, and reduces the impact of watermark embedding on the sound quality of each short audio. Unlike the usual music information retrieval that locks the complete music, this scheme can also segment a piece of music through technical means, and then perform information retrieval on each segment to extract the strategy and content of its production protection information. In addition, by uniformly interweaving the protection information of different creative contents, only the platform party can master the combination method, which can effectively hide the content of the protection information and reduce the possibility of the digital watermark being cracked.

[0101] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0102] Alternatively, the present invention may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0103] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0104] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An audio data processing method, comprising: Generate a watermark code representing the characteristics of the audio data; embedding a portion of the watermark code into each of the plurality of audio data as a digital watermark, The audio data is short audio data that is cited multiple times or appears together in a piece of music. Two or more digital watermarks of the audio data can be extracted to restore the watermark code, and the number of different digital watermarks required to restore the watermark code is determined according to the rules.

2. The method of claim 1, wherein: The audio data includes: Audio clips; sound effects clips; chord fragments; and / or Beat fragments.

3. The method of claim 1, further comprising: Generate copyright protection information for one or a group of audio data.

4. The method of claim 3, wherein: Generating a watermark encoding that characterizes the audio data includes: Based on the copyright protection information, the watermark code characterizing the characteristics of the one or a group of audio data is generated.

5. The method of claim 3, wherein: Generating a watermark encoding that characterizes the audio data includes: Based on the plurality of copyright protection information, the watermark code representing the characteristics of a plurality or a plurality of groups of audio data is generated.

6. The method of claim 5, wherein: Based on the plurality of copyright protection information, generating the watermark code representing the characteristics of the plurality or plurality of groups of audio data comprises: The watermark code representing the characteristics of multiple or multiple groups of audio data is generated through interleaving coding.

7. The method of claim 1, wherein: Embedding a portion of the watermark code into a plurality of audio data as a digital watermark respectively comprises: A portion of the watermark code is respectively embedded into a plurality of audio data copies of one audio data as a digital watermark.

8. The method of claim 1, wherein: Generating a watermark encoding that characterizes the audio data includes: Generate a watermark code that characterizes a musical work. Embedding a portion of the watermark code into a plurality of audio data as a digital watermark respectively comprises: A portion of the watermark code is embedded as a digital watermark into each of the plurality of audio data constituting the one musical piece.

9. The method of claim 1, further comprising: Get the audio data called by the creation entity. The generation of a watermark code representing the characteristics of the audio data includes: Generate a unified watermark encoding of the audio data representing the call; Embedding a portion of the watermark code into a plurality of audio data as a digital watermark respectively comprises: A portion of the unified watermark code is respectively embedded in the called audio clips in the music work created by the creative entity as a digital watermark.

10. The method of claim 1, further comprising: The watermark code and / or copyright protection information used to generate the watermark code is published.

11. The method of claim 10, wherein: The release includes: Announce to the public through the platform; together with the publication of the time of generation of said watermark encoding and / or said copyright protection information; or Published using blockchain.

12. The method of claim 1, further comprising: Publish multiple audio data through the music creation platform.

13. The method of claim 1, further comprising: The plurality of audio data are input into a retrieval library.

14. A method for processing audio and video data, comprising: Generate a watermark code that represents the characteristics of audio and video data; embedding a portion of the watermark code into a plurality of audio and video data, each of which includes audio data and image frame data, as a digital watermark, The audio data is short audio data that is cited multiple times or appears together in a piece of music. The digital watermarks of two or more audio and video data can be extracted to restore the watermark code, and the number of different digital watermarks required to restore the watermark code is determined according to the rules.

15. The method of claim 14, wherein: Embedding a portion of the watermark code into a plurality of audio and video data as a digital watermark comprises: Embedding a portion of the watermark code into the audio data and the image frame data included in the audio and video data as a digital watermark; A portion of the watermark code is associated and embedded into audio data and image frame data included in the audio and video data as a digital watermark.

16. A music composition method, comprising adding the audio data as claimed in any one of claims 1 to 13 or the audio data or image frame data as claimed in any one of claims 14 to 15 to generate a music work embedded with the digital watermark.

17. The method of claim 16, further comprising: Publish the musical work created.

18. A digital watermark extraction method, comprising: Divide the acquired music piece into multiple audio segments to be detected; Determine whether the multiple audio segments to be detected contain the audio data embedded with the digital watermark according to any one of claims 1 to 13; A digital watermark is extracted from the audio segment to be detected that contains the audio data.

19. The method of claim 18, further comprising: In the case where two or more different digital watermarks embedded in the audio data are extracted, the watermark code is restored from the multiple different digital watermarks.

20. The method of claim 19, further comprising: The copyright protection information used to generate the watermark code is recovered from the watermark code.

21. The method of claim 18, wherein: Determining whether the multiple audio segments to be detected contain the audio data embedded with the digital watermark according to any one of claims 1 to 13 comprises: Searching a search library including the audio data; and According to the search hit result, it is determined whether the multiple audio segments to be detected contain audio data embedded with the digital watermark.

22. A method for extracting watermarks from streaming media, comprising: Get streaming data; Detecting whether the streaming media data contains the audio data embedded with the digital watermark according to any one of claims 1 to 13 or the audio data or image frame data embedded with the digital watermark according to any one of claims 14 to 15, segment by segment; extracting a digital watermark contained in a segment of the audio data or the image frame data; as well as In the case that a plurality of different digital watermarks embedded in two or more of the audio data or the image frame data are extracted, the watermark code is restored from the plurality of different digital watermarks.

23. The method of claim 22, wherein: The streaming media includes: The video or short video posted; Live content; and / or Cloud interactive content.

24. A method for adding a watermark to streaming media, comprising adding the audio data described in any one of claims 1-13 or the audio data or image frame data described in any one of claims 14-15 to generate a streaming media work embedded with the digital watermark.

25. The method of claim 24, wherein: The streaming media includes: The video or short video posted; Live content; and / or Cloud interactive content.

26. An audio data processing device, comprising: A code generation unit, used to generate a watermark code representing the characteristics of the audio data; a watermark embedding unit, configured to embed a portion of the watermark code into each of the plurality of audio data as a digital watermark, The audio data is short audio data that is cited multiple times or appears together in a piece of music. Two or more digital watermarks of the audio data can be extracted to restore the watermark code, and the number of different digital watermarks required to restore the watermark code is determined according to the rules.

27. A digital watermark extraction device, comprising: A segmentation unit, used for dividing the acquired music piece into a plurality of audio segments to be detected; a matching unit, configured to determine whether the plurality of audio segments to be detected contain the audio data embedded with the digital watermark according to any one of claims 1 to 13; as well as The extraction unit is used to extract the digital watermark in the audio segment to be detected that contains the audio data.

28. The apparatus of claim 27, further comprising: A synthesis unit is used to restore the watermark code from the multiple different digital watermarks when extracting multiple different digital watermarks embedded in two or more of the audio data.

29. A music collaborative creation system, comprising a server and multiple clients, in, The server is used to: Providing a plurality of audio data processed by the method according to any one of claims 1 to 13; The client is used to: Using one or more of the plurality of audio data to perform music collaborative creation; as well as The created musical work is published, wherein the musical work includes one or more of the digital watermarks.

30. The system of claim 29, wherein: The client is also used to: Create and publish audio data, The server is used to: Use the method described in any one of claims 1-13 to add a digital watermark to audio data created and published by a client.

31. The system of claim 29, wherein: The server is also used to: For the acquired music work, the digital watermark extraction method as described in any one of claims 18 to 21 is executed.

32. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 25.

33. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 25.

Images