Multi-modal content copyright authentication and tracking method based on large model and block chain

Abstract

The invention relates to the field of data processing, and discloses a multi-modal content copyright authentication and tracking method based on a large model and a block chain. Comprising the following steps: acquiring text, image, audio or video content and right declaration information, normalizing the content and calculating an identity encryption abstract value; calling a large model to obtain a semantic fingerprint and generate a robust fingerprint, constructing an evidence packet and calculating an evidence abstract, and generating a digital signature by using a private key; the evidence packet is encrypted and stored under the chain to obtain an under-chain pointer, and the intelligent contract generates a work identifier and writes the work identifier into an on-chain evidence storage record; in the tracking stage, fingerprints are generated for target content, works are retrieved, matched work identifiers are determined according to threshold values, tracking evidences are output, and the tracking evidences comprise on-chain transaction identifiers, block heights, timestamps, signature verification results and off-chain pointers.

Description

Technical Field

[0001] This invention relates to the field of data processing, specifically to a method for multimodal content copyright authentication and tracking based on large models and blockchain. Background Technology

[0002] Against the backdrop of widespread internet content distribution and short video platforms, works are frequently published, reproduced, and redistributed in the forms of text, images, audio, and video, often accompanied by editing, splicing, compression, subtitle replacement, dubbing mashups, and semantic rewriting. At the same time, generative large-scale models lower the barrier to cross-modal re-creation, placing higher demands on rights holders regarding the timeliness and verifiability of rights confirmation, authorization management, and infringement evidence collection.

[0003] Existing evidence preservation methods typically employ file hashing combined with timestamp services or blockchain records to achieve time verification, supplemented by electronic signatures, trusted time sources, and third-party evidence preservation platforms. For tracking and identification, common solutions include digital watermarking, perceptual hashing, audio / video fingerprinting, and feature database comparison; some solutions introduce deep feature vectors for similar content retrieval. However, most solutions focus on a single modality or can only handle format-level changes, lacking the ability to correlate cross-modal rewrites. Furthermore, the binding granularity between on-chain records and off-chain evidence objects is inconsistent, and the model version and feature extraction configuration lack traceability and consistency, affecting the consistency of evidence verification.

[0004] Existing technologies struggle to establish a unified, reproducible, verifiable, and strictly interconnected on-chain and off-chain evidence chain for evidence storage and tracking, even when works undergo compression, transcoding, cropping, speed adjustment, or semantic rewriting and are disseminated across modalities. This makes it difficult to meet the verifiability requirements of evidence in infringement dispute resolution. Therefore, we propose a multimodal content copyright authentication and tracking method based on a large-scale model and blockchain. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal content copyright authentication and tracking method based on a large model and blockchain. This method solves the technical problems in existing technologies, such as inconsistencies in field meanings and value specifications of multi-source customer feedback data, difficulty in uniformly cleaning and merging duplicate and abnormal records, difficulty in reproducing stable attribution and KPI standards due to the time-sensitive changes in dealer hierarchy relationships, and difficulty in balancing result consistency and response efficiency due to repeated calculations required during the query phase.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: A multimodal content copyright authentication and tracking method based on large-scale models and blockchain includes the following steps: S1: Obtain the content data to be stored and the rights statement information corresponding to the content data, wherein the content data is one of text, image, audio or video; S2: Perform normalization processing according to the modal type of the content data to obtain normalized content, and calculate the identity encryption digest value for the normalized content; S3: Invoke the multimodal large model to generate a cross-modal semantic representation for the normalized content, and convert the cross-modal semantic representation into a fixed-length semantic fingerprint based on the quantization rule parameters; S4: Generate a robust fingerprint based on the normalized content according to the modal type. The robust fingerprint is used to maintain stable matching under content changes caused by compression, transcoding, cropping, speed change or rewriting. S5: Construct an evidence package, which includes at least the identity encryption digest value, the semantic fingerprint, the robust fingerprint, the rights statement information, the model version identifier of the multimodal large model, and the feature extraction configuration; S6: Calculate an evidence digest for the evidence package and generate a digital signature for the evidence digest using the rights holder's private key; S7: Store the evidence package or its encrypted copy in an off-chain storage medium to obtain an off-chain pointer, and calculate the pointer digest of the off-chain pointer; S8: Invoke the blockchain smart contract to generate a work identifier and write it into the evidence storage record. The evidence storage record includes at least the evidence digest, semantic fingerprint digest, robust fingerprint digest, pointer digest, timestamp, and digital signature. The semantic fingerprint digest is a digest value obtained by performing a cryptographic hash operation on the semantic fingerprint, and the robust fingerprint digest is a digest value obtained by performing a cryptographic hash operation on the robust fingerprint. S9: Establish a cross-modal tracking index, associate the work identifier with the semantic fingerprint, and use it to perform nearest neighbor retrieval in the cross-modal tracking index; S10: During the tracking phase, the target content is acquired and normalized to generate a semantic fingerprint and a robust fingerprint of the target content; a candidate work set is recalled based on the semantic fingerprint, and a semantic similarity index and a robust matching index are calculated for the candidate works. The matching work identifier is determined according to the threshold parameter; tracking evidence is output, which includes at least the on-chain transaction identifier, block height, timestamp, digital signature verification result, and off-chain pointer corresponding to the matching work identifier.

[0007] Preferably, a registration request is received through a registration interface to obtain the content data and the rights statement information, wherein the registration request includes content subject data and content metadata; The content metadata includes modality type identifier, data format identifier, encoding method identifier, and data length field; The rights statement information includes the rights holder's identity identifier, rights type identifier, scope of license field, license period field, statement generation time field, and rights holder's public key identifier.

[0008] Preferably, the normalization process converts the content data into a normalized byte stream, which is obtained by concatenating a modality type identifier, a data format identifier, an encoding method identifier, and a normalized content payload in a predetermined order. The identity cryptographic digest value is obtained by performing a fixed-length cryptographic hash operation on the normalized byte stream, and the hash algorithm identifier is associated with and stored with the identity cryptographic digest value.

[0009] Preferably, when the content data is text, the text is decoded and transcoded into a unified character encoding, and after performing Unicode normalization and unifying the newline character, it is serialized into a text payload; When the content data is an image, the image is decoded into a pixel matrix and the orientation is corrected. It is then converted into a unified color space and a unified bit depth, and serialized into an image payload in row priority order. When the content data is audio, the audio is decoded into a sampling sequence and the channel order is corrected. After unifying the sampling format and byte order, it is serialized into an audio payload in chronological order. When the content data is video, the video frame sequence and audio sampling sequence are parsed. The video frame sequence is decoded into a unified color space and a unified bit depth in the order of presentation time and then serialized into a video payload. The audio sampling sequence is serialized according to the serialization rules of the audio payload.

[0010] Preferably, the multimodal large model includes a text encoding sub-model, an image encoding sub-model, an audio encoding sub-model, and a video encoding sub-model, with the encoding outputs of different modalities mapped to the same semantic vector space; the cross-modal semantic representation is a fixed-length vector, and the dimension of the fixed-length vector is associated with the feature extraction configuration and stored.

[0011] Preferably, converting the cross-modal semantic representation into a fixed-length semantic fingerprint includes: performing normalization processing on the cross-modal semantic representation; Based on the linear projection matrix determined by the feature extraction configuration, a linear projection is performed on the normalized vector; The projected vector is uniformly quantized according to the quantization bit width determined by the feature extraction configuration to obtain quantized codewords, and the quantized codewords are encapsulated according to a predetermined byte order to obtain the semantic fingerprint.

[0012] Preferably, the robust fingerprint is generated by: determining robust feature extraction rules based on the feature extraction configuration, and extracting a robust feature set from the normalized content; Calculate the feature hash value for each robust feature and sort them in ascending order; A predetermined number of feature hash values ​​are selected after sorting to form a robust signature, and the robust signature is encapsulated in a predetermined byte order to obtain the robust fingerprint. The predetermined number and the robust feature extraction rule are limited by the feature extraction configuration.

[0013] Preferably, the evidence package is a structured data object and includes: a content identifier field, a rights statement field, a fingerprint field, a model version field, and a configuration version field; The content identifier field includes the request identifier, the identity encryption digest value, and its algorithm identifier; The fingerprint field includes the semantic fingerprint and its length identifier, and the robust fingerprint and its length identifier; The configuration version field consists of the encrypted digest value calculated after the feature extraction configuration is serialized according to a predetermined serialization rule.

[0014] Preferably, calculating the evidence digest for the evidence package includes performing deterministic serialization of the evidence package in a predetermined field order and performing a cryptographic hash operation on the serialized data; The digital signature is a signature value obtained by performing an asymmetric cryptographic signature operation on the evidence digest using the right holder's private key, and the signature algorithm identifier, the signature value, and the right holder's public key identifier are written into the evidence package.

[0015] Preferably, storing the evidence package or its encrypted copy in an off-chain storage medium includes performing symmetric encryption on the evidence package to obtain ciphertext of the evidence package and generating an object identifier, and writing the object identifier and the ciphertext of the evidence package into the off-chain storage medium to obtain an off-chain pointer; The pointer digest is obtained by performing a cryptographic hash operation on the off-chain pointer after deterministically serializing it according to a predetermined field order; When the blockchain smart contract is invoked to generate a work identifier and write it into the evidence storage record, the evidence digest, the semantic fingerprint digest, the robust fingerprint digest, the pointer digest, the timestamp, and the digital signature are written into the evidence storage record and an on-chain transaction identifier is returned. The cross-modal tracking index includes a vector index library and a mapping table. The vector index library is used to recall a set of candidate work identifiers based on the retrieval vector corresponding to the target content. The mapping table is used to associate the candidate work identifiers with the on-chain transaction identifiers. When determining the matching work identifier, the threshold parameters corresponding to the semantic similarity index calculated based on the retrieval vector and the robust matching degree index obtained based on the robust fingerprint consistency comparison are simultaneously satisfied; when outputting tracking evidence, the tracking evidence includes the on-chain transaction identifier, block height, timestamp, digital signature verification result, and off-chain pointer corresponding to the matching work identifier.

[0016] In summary, the present invention has the following main beneficial effects: By performing unified normalization processing on content of different modalities and constructing normalized byte streams, and then calculating identity encryption digest values ​​on the normalized byte streams to achieve a deterministic representation of content identity, the same work can obtain consistent digest results under the same normalization rules. This provides a unified basic identifier for evidence storage records, evidence packages and subsequent tracking and verification, and reduces the risk of inconsistency in evidence storage caused by differences in encoding, format or payload organization.

[0017] By calling a multimodal large model to generate cross-modal semantic representations and generating fixed-length semantic fingerprints under feature extraction configuration constraints, and generating robust fingerprints according to modality type, a dual representation of semantic consistency and content robustness is achieved. This ensures that the target content can still form a stable retrieval and matching basis under cross-modal rewriting or content changes such as compression, transcoding, cropping, and speed variation, thereby improving the consistency of candidate recall and the verifiability of matching judgment in the tracking stage.

[0018] By constructing an evidence package containing a unique encrypted digest value, semantic fingerprint, robust fingerprint, rights statement information, model version identifier, and feature extraction configuration, calculating an evidence digest for the evidence package and generating a digital signature using the rights holder's private key, and then encrypting and storing the evidence package off-chain, while writing the evidence digest, fingerprint digest, and pointer digest into the on-chain evidence storage record, the collaboration between on-chain immutable anchoring and off-chain evidence object positioning is achieved. This enables third parties to complete time and signature verification based on on-chain transaction identifiers, block heights, and timestamps, and to locate the evidence package based on off-chain pointers, thereby enhancing the verifiability and consistency of evidence storage and tracking. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1 refer to Figure 1 A multimodal content copyright authentication and tracking method based on large-scale models and blockchain, comprising the following steps: S1: Obtain the content data to be stored and the rights statement information corresponding to the content data, wherein the content data is one of text, image, audio or video; S2: Perform normalization processing according to the modal type of the content data to obtain normalized content, and calculate the identity encryption digest value for the normalized content; S3: Invoke the multimodal large model to generate a cross-modal semantic representation for the normalized content, and convert the cross-modal semantic representation into a fixed-length semantic fingerprint based on the quantization rule parameters; S4: Generate a robust fingerprint based on the normalized content according to the modal type. The robust fingerprint is used to maintain stable matching under content changes caused by compression, transcoding, cropping, speed change or rewriting. S5: Construct an evidence package, which includes at least the identity encryption digest value, the semantic fingerprint, the robust fingerprint, the rights statement information, the model version identifier of the multimodal large model, and the feature extraction configuration; S6: Calculate an evidence digest for the evidence package and generate a digital signature for the evidence digest using the rights holder's private key; S7: Store the evidence package or its encrypted copy in an off-chain storage medium to obtain an off-chain pointer, and calculate the pointer digest of the off-chain pointer; S8: Invoke the blockchain smart contract to generate a work identifier and write it into the evidence storage record. The evidence storage record includes at least the evidence digest, semantic fingerprint digest, robust fingerprint digest, pointer digest, timestamp, and digital signature. The semantic fingerprint digest is a digest value obtained by performing a cryptographic hash operation on the semantic fingerprint, and the robust fingerprint digest is a digest value obtained by performing a cryptographic hash operation on the robust fingerprint. S9: Establish a cross-modal tracking index, associate the work identifier with the semantic fingerprint, and use it to perform nearest neighbor retrieval in the cross-modal tracking index; S10: During the tracking phase, the target content is acquired and normalized to generate a semantic fingerprint and a robust fingerprint of the target content; a candidate work set is recalled based on the semantic fingerprint, and a semantic similarity index and a robust matching index are calculated for the candidate works. The matching work identifier is determined according to the threshold parameter; tracking evidence is output, which includes at least the on-chain transaction identifier, block height, timestamp, digital signature verification result, and off-chain pointer corresponding to the matching work identifier.

[0022] The registration request is received through the registration interface to obtain content data and rights statement information. The registration request includes a byte sequence of the main content data. With content metadata and include a rights statement. The system generates a registration request identifier. And form a registration record: ;in, Used to link evidence packages, off-chain pointers, and on-chain records; content metadata is defined as... ;in, Modal type identifier, with values ​​limited to: These correspond to text, images, audio, and video, respectively. Data format identifier, used to indicate the format category of content data; Encoding method identifier, used to indicate the encoding or decoding method used for the content data; The data length field satisfies ; byte sequence The length in bytes.

[0023] Rights statement information is defined as ;in, As an identifier for the rights holder; Identifier for the type of right; This is a permission scope field; For license period field; To declare and generate a time field; The public key identifier for the rights holder is used to indicate the public key participating in subsequent verification; the scope of license field and the license period field are further defined as follows: ;in, For registration range subfields; To use range subfields; For derived range subfields; ;in, The start time of the license; License expiration time, meets ≤ .

[0024] The registration interface performs consistency checks on registration requests, including at least the following: and When the verification fails, the registration interface refuses to generate a registration record and returns the corresponding verification failure reason identifier.

[0025] Define a deterministic serialization function Byte sequence concatenation operation ; For identifier fields It is encoded as a byte sequence with a length prefix.

[0026] For byte sequence fields The output is generated by adding a length prefix to the original byte sequence.

[0027] splicing operation This indicates that two byte sequences are concatenated in sequential order to obtain a new byte sequence.

[0028] Content body data byte sequence By modal type Perform normalization processing to obtain normalized content payload. ;in, A normalized function is a deterministic function; under the same conditions... , , , Under the same normalization rules, the output Maintain consistency.

[0029] The modal rule is as follows: when At that time, according to Decode into character sequence Executing Unicode standardization yields By standardizing the newline character to a fixed format, we obtain Then output the byte sequence according to the unified character encoding: ;in, Depend on The character sequence obtained after decoding; The character sequence after Unicode normalization; The character sequence after unifying the newline character; For unified character encoding identification; This is an encoding function that converts a character sequence into a byte sequence according to a specified character encoding. when At that time, Decoded into pixel matrix ; Execution direction correction obtained Converted to a unified color space and with a unified bit depth, resulting in Flatten and output in row priority and fixed channel order: ;in, It is a pixel matrix; This is the pixel matrix after orientation correction; A pixel matrix after unifying color space and bit depth; The flattening function flattens a matrix into a sequence of bytes in a predetermined order; when At that time, Decode the sampled sequence, correct the channel order, unify the sampling format and byte order, and then serialize and output it in chronological order: ;in, This is an audio serialization function that outputs a normalized audio byte sequence arranged in chronological order; when At that time, the video frame sequence and audio sampling sequence are obtained through parsing. After the frame sequence is sorted in ascending order by presentation timestamp, each frame undergoes unified processing consistent with the image modality and is serialized and spliced ​​to form the frame payload. The audio sampling sequence forms the audio payload. Combined in a fixed order: ;in, This refers to the number of video frames. For the first The frame payload byte sequence of the frame is generated according to the image modality normalization rules; This is the audio payload byte sequence of the video audio track.

[0030] The modality type identifier, data format identifier, encoding method identifier, and normalized content payload are concatenated in a predetermined order to obtain a normalized byte stream. ;in, To standardize byte streams; , , , They are respectively for , , , Deterministic serialization output.

[0031] right Perform a fixed-length cryptographic hash operation to obtain a cryptographic digest value with identity. ;in, It is a cryptographic hash function with a fixed output length; and it forms a digest associated record. ;in, Hash algorithm identifier, used to indicate the computation The hash algorithm used; Associate records with the summary.

[0032] The multimodal large model includes text coding sub-models, image coding sub-models, audio coding sub-models, and video coding sub-models. This applies to normalized content payloads. Generate cross-modal semantic representation vectors And map the outputs of different modalities to the same semantic vector space: ;in, It is a cross-modal semantic representation vector; For dimension The real vector space; It is a semantic vector dimension, associated with the feature extraction configuration and stored.

[0033] This embodiment records the model version identifier. and feature extraction configuration Version identifier The configuration version identifier is defined as follows: ;in, For feature extraction configuration, at least include Parameters and rule identifiers. Configure a version identifier to anchor the conditions for this fingerprint generation. Let be the linear projection matrix and the bias vector. Projection dimension. Quantization bit width. : Quantization scale parameter. : Number of robust signatures selected. : Robust feature hash output bit length.

[0034] Normalization processing, for Performing L2 normalization yields : ;in, : Normalized semantic vector; The 2-norm.

[0035] Linear projection based on configuration The projection matrix in With bias Performing linear projection yields ;in, : Projection vector, dimension is ; Projection matrix; Bias vector; Projection dimension.

[0036] Uniform quantization yields a quantized codeword with a quantization bit width of 1. Quantization series .right The Dimensional components First, cut off: ;in, : The One component; The truncated component; : Truncation function, restricts the input to a range Inside; : Quantization scale parameter.

[0037] Calculate the quantized codewords again : ; in : No. Each quantized codeword has a value range of 1. ; : Quantization series, satisfying ; Quantization bit width; : Floor operation; Encapsulates the codeword sequence into a semantic fingerprint according to a fixed byte order. And record its length. For subsequent on-chain evidence storage, calculate the semantic fingerprint digest: ;in, Semantic fingerprint byte sequence; Semantic fingerprint byte length; Semantic fingerprint summary value.

[0038] Based on configuration Determine robust feature extraction rules From the perspective of standardized content load Extracting robust feature sets: ; in, Robust feature extraction rule identifiers and parameter sets; Robust feature extraction function; Robust feature set; : No. One robust feature; : Number of robust features; Calculate the feature hash value for each robust feature: ;in, : Feature hash function, output bit length is determined by limited; : No. Each feature hash value; : Feature hash output bit length; The sequence is obtained by sorting the values ​​in ascending order. Select the first sorted items A robust signature is formed and encapsulated as a robust fingerprint. For subsequent on-chain evidence storage, a robust fingerprint digest is calculated: ;in, : Select the quantity, which is a positive integer, from limited; Robust fingerprint byte sequence; : Robust fingerprint digest value.

[0039] Construct a structured evidence package, which includes at least an identity cryptographic digest value, a semantic fingerprint, a robust fingerprint, a rights statement, a model version identifier, and a feature extraction configuration version identifier. ; in; Evidence package; Model version identifier; To ensure the recalculation of the evidence summary, this embodiment specifies the deterministic serialization function for the evidence package. The field order is fixed as follows: ; in, : A function that performs deterministic serialization on the evidence package.

[0040] For evidence packages that do not contain a signature field Perform deterministic serialization and hash to obtain evidence digest : ; in, : Evidence package version that does not contain signature-related fields; Evidence digest value; using the rights holder's private key Generate digital signatures from evidence digests : ; in, : The rights holder's private key; : Signature algorithm identifier; Signature algorithm; : Signature value; identifies the signature algorithm Signature value With public key identifier Write the evidence package to form a signed evidence package : ; in, : A package containing signed evidence.

[0041] Off-chain encrypted storage and off-chain pointer generation pairs Perform deterministic serialization to obtain the byte sequence of the evidence packet. : ;in; : The byte sequence after serialization of the evidence packet; for Perform symmetric encryption to obtain ciphertext : ;in, : Identifier of symmetric encryption algorithm; Symmetric key; : Initial vector or random variable; Symmetric encryption function; : Evidence package encrypted text; Identifier of the object to be calculated from the encrypted text : ;in, : Object identifier, used to locate objects in the chain; Write to off-chain storage medium and obtain off-chain pointer ;in, : Off-chain storage medium identifier; : Off-chain pointer.

[0042] Calculate pointer digest for off-chain pointers ;in, Pointer digest value; to ensure that off-chain pointers can be output during the tracing phase; a pointer mapping library is established during the evidence storage phase. ,Will Write So that in the tracking phase, based on Reverse lookup yields the corresponding result The on-chain evidence storage record writing and transaction identifier return process writes the evidence digest, semantic fingerprint digest, robust fingerprint digest, pointer digest, timestamp, and digital signature into the on-chain evidence storage record. This constructs the on-chain evidence storage record. : ; in, On-chain evidence storage records; : Record timestamps on the blockchain; invoke smart contracts to write to the blockchain and return the artwork identifier and on-chain transaction identifier: ; in, The process of invoking a smart contract; Work identifier; : On-chain transaction identifier.

[0043] After block confirmation, based on transaction identifier The query retrieves the block height. With timestamp : ; in, On-chain query process; Block height.

[0044] Cross-modal tracking index establishment: This embodiment establishes a cross-modal tracking index, including a vector index library and a mapping table. The vector index library is used to retrieve vectors. As an index item, the work identifier is associated with the retrieval vector: ;in, Vector index library; : Index entries; the mapping table is used to associate work identifiers with on-chain transaction identifiers: ;in, : Mapping table.

[0045] Candidate recall during the tracking phase, dual-indicator threshold determination, and tracking evidence output; acquisition of target content byte sequence during the tracking phase. and its content metadata Generate target retrieval vectors according to the same rules as in Parts 3 to 6. Semantic fingerprint With robust fingerprints In the vector index library... Perform nearest neighbor retrieval to recall candidate set For any candidate entry Calculate semantic similarity index : ; in, : No. One candidate semantic similarity index; : The retrieval vector corresponding to the candidate work; unpacking the robust fingerprint into a set of hash values and Calculate the robust matching index : ; in, : No. Robust matching metrics for each candidate; : Robust hash set of target content; Robust hash set of candidate works; : The number of elements in the set.

[0046] Set threshold parameters If and only if the following condition is met: ; in, Semantic similarity threshold parameter; Robust matching threshold parameter; determines the identifier of the matching work. Then, the corresponding transaction identifier is obtained through the mapping table. And obtain the block height and timestamp through on-chain queries. According to the records on the chain Perform the signature verification to obtain the verification result. : ; in, Signature verification algorithm; : The rights holder's public key; The signature verification result is either "pass" or "fail"; and it is based on the records in the blockchain. via pointer mapping library Reverse lookup yields the off-chain pointer. The final output is tracking evidence. : ; in, Tracking evidence; Match the work's identifier; : Match the on-chain transaction identifier corresponding to the work; : Match the block height corresponding to the transaction; : Match the timestamp corresponding to the transaction; : Matches the off-chain pointer corresponding to the work.

[0047] During the evidence preservation stage, the registration interface receives registration requests and generates registration records, using the registration request identifier to consistently associate the content subject data, content metadata, and rights statement information. Subsequently, the system performs submodal normalization processing on the content subject data based on the modality type, data format, and encoding method in the content metadata, obtaining a normalized content payload. A normalized byte stream is constructed under unified deterministic serialization and concatenation rules, and then an identity cryptographic digest value is calculated to characterize the identity of the content under the normalization rules. The system calls a multimodal large model to generate cross-modal semantic representations of the normalized content, and converts these semantic representations into fixed-length semantic fingerprints under feature extraction configuration constraints. Simultaneously, a robust fingerprint is generated based on robust feature extraction rules matching the modality type. These fingerprints, along with the rights statement information, model version identifier, and feature extraction configuration, constitute an evidence package. After deterministic serialization, the evidence package calculates an evidence digest, and a digital signature is generated using the rights holder's private key to ensure that a third party can verify the consistency between the evidence digest and the signature based on the public key. The evidence package or its encrypted copy is written to off-chain storage to obtain an off-chain pointer, and a pointer digest is calculated from the off-chain pointer. Simultaneously, the smart contract generates a work identifier and writes it to an on-chain notarization record. This on-chain notarization record includes at least an evidence digest, a semantic fingerprint digest, a robust fingerprint digest, a pointer digest, a timestamp, and a digital signature, thus forming immutable on-chain anchoring information and a locatable off-chain evidence object. The system further establishes a cross-modal tracking index, associating the work identifier with the semantic fingerprint and using it for nearest neighbor retrieval. During the tracking phase, the target content undergoes the same normalization and feature extraction process as in the notarization phase to obtain the target semantic fingerprint and the target robust fingerprint. First, a set of candidate works is recalled from the tracking index based on the semantic fingerprint. Then, the candidate works are jointly judged based on semantic similarity and robust matching degree to determine the matching work identifier. Finally, tracking evidence is output, including the on-chain transaction identifier, block height, timestamp, digital signature verification result, and off-chain pointer corresponding to the matching work identifier. This allows third parties to verify the time and signature based on the on-chain record and locate the corresponding evidence package content based on the off-chain pointer.

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for multimodal content copyright authentication and tracking based on large-scale models and blockchain, characterized in that, Includes the following steps: S1: Obtain the content data to be stored and the rights statement information corresponding to the content data, wherein the content data is one of text, image, audio or video; S2: Perform normalization processing according to the modal type of the content data to obtain normalized content, and calculate the identity encryption digest value for the normalized content; S3: Invoke the multimodal large model to generate a cross-modal semantic representation for the normalized content, and convert the cross-modal semantic representation into a fixed-length semantic fingerprint based on the quantization rule parameters; S4: Generate a robust fingerprint based on the normalized content according to the modal type. The robust fingerprint is used to maintain stable matching under content changes caused by compression, transcoding, cropping, speed change or rewriting. S5: Construct an evidence package, which includes at least the identity encryption digest value, the semantic fingerprint, the robust fingerprint, the rights statement information, the model version identifier of the multimodal large model, and the feature extraction configuration; S6: Calculate an evidence digest for the evidence package and generate a digital signature for the evidence digest using the rights holder's private key; S7: Store the evidence package or its encrypted copy in an off-chain storage medium to obtain an off-chain pointer, and calculate the pointer digest of the off-chain pointer; S8: Invoke the blockchain smart contract to generate a work identifier and write it into the evidence storage record. The evidence storage record includes at least the evidence digest, semantic fingerprint digest, robust fingerprint digest, pointer digest, timestamp, and digital signature. The semantic fingerprint digest is a digest value obtained by performing a cryptographic hash operation on the semantic fingerprint, and the robust fingerprint digest is a digest value obtained by performing a cryptographic hash operation on the robust fingerprint. S9: Establish a cross-modal tracking index, associate the work identifier with the semantic fingerprint, and use it to perform nearest neighbor retrieval in the cross-modal tracking index; S10: During the tracking phase, the target content is acquired and normalization processing is performed to generate the semantic fingerprint and robust fingerprint of the target content; Based on the semantic fingerprint, a set of candidate works is recalled, and semantic similarity index and robust matching index are calculated for the candidate works. The identifier of the matching work is determined according to the threshold parameter. Output tracking evidence, which includes at least the on-chain transaction identifier corresponding to the matching work identifier, the block height, the timestamp, the digital signature verification result, and the off-chain pointer.

2. The method for multimodal content copyright authentication and tracking based on a large model and blockchain as described in claim 1, characterized in that, The registration request is received through the registration interface to obtain the content data and the rights statement information. The registration request includes content subject data and content metadata. The content metadata includes modality type identifier, data format identifier, encoding method identifier, and data length field; The rights statement information includes the rights holder's identity identifier, rights type identifier, scope of license field, license period field, statement generation time field, and rights holder's public key identifier.

3. The method for multimodal content copyright authentication and tracking based on a large model and blockchain as described in claim 2, characterized in that, The normalization process converts the content data into a normalized byte stream, which is obtained by concatenating modality type identifier, data format identifier, encoding method identifier, and normalized content payload in a predetermined order. The identity cryptographic digest value is obtained by performing a fixed-length cryptographic hash operation on the normalized byte stream, and the hash algorithm identifier is associated with and stored with the identity cryptographic digest value.

4. The method for multimodal content copyright authentication and tracking based on a large model and blockchain as described in claim 3, characterized in that, When the content data is text, the text is decoded and transcoded into a unified character encoding, and after performing Unicode normalization and unifying the newline character, it is serialized into a text payload. When the content data is an image, the image is decoded into a pixel matrix and the orientation is corrected. It is then converted into a unified color space and a unified bit depth, and serialized into an image payload in row priority order. When the content data is audio, the audio is decoded into a sampling sequence and the channel order is corrected. After unifying the sampling format and byte order, it is serialized into an audio payload in chronological order. When the content data is video, the video frame sequence and audio sampling sequence are parsed. The video frame sequence is decoded into a unified color space and a unified bit depth in the order of presentation time and then serialized into a video payload. The audio sampling sequence is serialized according to the serialization rules of the audio payload.

5. The method for multimodal content copyright authentication and tracking based on a large model and blockchain as described in claim 4, characterized in that, The multimodal large model includes a text encoding sub-model, an image encoding sub-model, an audio encoding sub-model, and a video encoding sub-model. The encoding outputs of different modalities are mapped to the same semantic vector space. The cross-modal semantic representation is a fixed-length vector, and the dimension of the fixed-length vector is associated with the feature extraction configuration and stored.

6. The method for multimodal content copyright authentication and tracking based on a large model and blockchain according to claim 5, characterized in that, The conversion of the cross-modal semantic representation into a fixed-length semantic fingerprint includes: performing normalization processing on the cross-modal semantic representation; Based on the linear projection matrix determined by the feature extraction configuration, a linear projection is performed on the normalized vector; The projected vector is uniformly quantized according to the quantization bit width determined by the feature extraction configuration to obtain quantized codewords, and the quantized codewords are encapsulated according to a predetermined byte order to obtain the semantic fingerprint.

7. The method for multimodal content copyright authentication and tracking based on a large model and blockchain as described in claim 6, characterized in that, The robust fingerprint is generated as follows: robust feature extraction rules are determined based on the feature extraction configuration, and a robust feature set is extracted from the normalized content; Calculate the feature hash value for each robust feature and sort them in ascending order; A predetermined number of feature hash values ​​are selected after sorting to form a robust signature, and the robust signature is encapsulated in a predetermined byte order to obtain the robust fingerprint. The predetermined number and the robust feature extraction rule are limited by the feature extraction configuration.

8. The method for multimodal content copyright authentication and tracking based on a large model and blockchain according to claim 7, characterized in that, The evidence package is a structured data object and includes: a content identifier field, a rights statement field, a fingerprint field, a model version field, and a configuration version field; The content identifier field includes the request identifier, the identity encryption digest value, and its algorithm identifier; The fingerprint field includes the semantic fingerprint and its length identifier, and the robust fingerprint and its length identifier; The configuration version field consists of the encrypted digest value calculated after the feature extraction configuration is serialized according to a predetermined serialization rule.

9. The method for multimodal content copyright authentication and tracking based on a large model and blockchain as described in claim 8, characterized in that, Calculating an evidence digest for the evidence package includes performing deterministic serialization of the evidence package in a predetermined field order and performing a cryptographic hash operation on the serialized data; The digital signature is a signature value obtained by performing an asymmetric cryptographic signature operation on the evidence digest using the right holder's private key, and the signature algorithm identifier, the signature value, and the right holder's public key identifier are written into the evidence package.

10. A method for multimodal content copyright authentication and tracking based on a large model and blockchain as described in claim 9, characterized in that, Storing the evidence package or its encrypted copy in an off-chain storage medium includes performing symmetric encryption on the evidence package to obtain ciphertext of the evidence package and generating an object identifier, and writing the object identifier and the ciphertext of the evidence package into the off-chain storage medium to obtain an off-chain pointer; The pointer digest is obtained by performing a cryptographic hash operation on the off-chain pointer after deterministically serializing it according to a predetermined field order; When the blockchain smart contract is invoked to generate a work identifier and write it into the evidence storage record, the evidence digest, the semantic fingerprint digest, the robust fingerprint digest, the pointer digest, the timestamp, and the digital signature are written into the evidence storage record and an on-chain transaction identifier is returned. The cross-modal tracking index includes a vector index library and a mapping table. The vector index library is used to recall a set of candidate work identifiers based on the retrieval vector corresponding to the target content. The mapping table is used to associate the candidate work identifiers with the on-chain transaction identifiers. When determining the matching work identifier, the threshold parameters corresponding to the semantic similarity index calculated based on the retrieval vector and the robust matching degree index obtained based on the robust fingerprint consistency comparison are simultaneously satisfied; when outputting tracking evidence, the tracking evidence includes the on-chain transaction identifier, block height, timestamp, digital signature verification result, and off-chain pointer corresponding to the matching work identifier.

Images