Copyright affiliation multi-dimensional automatic identification technology integrating deep learning and multi-modal intelligent analysis

Abstract

The invention provides a copyright ownership multi-dimensional automatic identification technology fusing deep learning and multi-modal intelligent analysis, and relates to the technical field of information processing, the copyright ownership multi-dimensional automatic identification technology comprises a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module, a classification identification module and a model optimization module, and all the modules work cooperatively to realize copyright ownership automatic identification. The data acquisition module is composed of a multi-modal data acquisition unit which is composed of an image acquisition unit, a text acquisition unit, an audio acquisition unit and a video acquisition unit, and is provided with corresponding acquisition equipment for acquiring, transmitting and storing multi-modal data according to a set process, and a multi-modal data preprocessing unit of the data preprocessing module can perform cleaning, format conversion, denoising and standardization operation on the data; the data quality is ensured, the copyright ownership identification accuracy is remarkably improved, data features are deeply mined through multi-modal fusion and a deep learning algorithm, complex copyright data are effectively processed, and the misjudgment risk is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of information processing technology, and more specifically, relates to a multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis. Background Art

[0002] In the field of copyright ownership identification, existing technologies have many limitations. Text recognition technology relies on keyword searches and text similarity comparisons, and faces challenges with keyword deformation and processing long, multilingual, and complex text structures. Image recognition technology faces numerous difficulties in processing complex data and addressing image tampering. Audio and video recognition technologies are weak in processing complex scenarios and are constrained by data volume. While existing technologies have been effective in single modalities, they lack multimodal fusion and struggle to integrate information from different modalities, resulting in inaccurate recognition. Furthermore, they require a high level of manual involvement, from feature annotation to threshold setting, reducing efficiency and accuracy. Processing capacity is insufficient in the face of large-scale data, and exclusive recognition is insufficient, making it impossible to distinguish highly similar copyright information, resulting in a lack of authoritative recognition results. Their simple system structure and linear process make it difficult to meet the needs of multimodal data fusion. They have significant shortcomings in processing multilingual text and complex multimedia data, creating an opportunity for improvement for the present invention. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides a multi-dimensional automatic identification technology of copyright ownership that integrates deep learning and multimodal intelligent analysis to solve the above problems.

[0004] The multi-dimensional automatic identification technology of copyright ownership integrates deep learning and multimodal intelligent analysis, which includes a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module, a classification and identification module, and a model optimization module. Each module works together to realize automatic identification of copyright ownership.

[0005] Preferably, the data acquisition module is composed of a multimodal data collector consisting of image, text, audio and video acquisition units, and is equipped with corresponding acquisition equipment to collect, transmit and store multimodal data according to the set process. The multimodal data preprocessing unit of the data preprocessing module can clean, convert the format, denoise and standardize the data to ensure data quality. The deep learning feature extractor of the feature extraction module adopts CNN, RNN and MFS models to extract the features of image video, text and audio respectively.

[0006] Preferably, the multimodal feature fusion network in the feature fusion module integrates multimodal features through feature-level fusion (such as splicing, weighted summation) and decision-level fusion (such as voting, weighting) to obtain the final classification result. The copyright ownership classifier of the classification and recognition module is based on a multi-task learning framework and predicts the copyright ownership of the fused features after training. The adaptive model optimizer of the model optimization module uses parameter updates, learning rate adjustment and regularization methods to improve model performance.

[0007] Preferably, the multimodal data deep fusion technology utilizes data complementarity through feature-level and decision-level fusion to improve the accuracy of copyright information extraction. Specifically, it adopts operations such as matrix splicing and voting. The automatic feature extraction algorithm automatically extracts key features of multi-dimensional data based on deep learning algorithms such as CNN, RNN, and MFS. CNN extracts image and video features through convolution pooling, RNN uses structures such as LSTM to capture text sequence features, and MFS calculates audio spectrum features. The multi-task learning framework collaboratively learns multiple tasks through a shared representation layer. In exclusive recognition, it uses a variety of similarity metrics and threshold setting methods to screen copyright information, thereby improving recognition efficiency and accuracy.

[0008] Compared with the prior art, the present invention has the following beneficial effects: Accurate identification: Significantly improve the accuracy of copyright identification. Multimodal fusion and deep learning algorithms deeply mine data features, effectively process complex copyright data, and reduce the risk of misjudgment.

[0009] Efficient processing: Greatly improves recognition efficiency. Automatic feature extraction and optimized model architecture reduce manual intervention and computing time. It can handle large-scale data and accelerate the copyright determination process.

[0010] Wide application: It has great application value in many fields such as copyright protection and content review, provides strong technical support for intellectual property protection, helps standardize digital content copyright management, and safeguards the rights and interests of creators. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a cross-sectional view of the technical architecture of the present invention; Figure 2 It is a flow chart of the copyright ownership identification system of the present invention. DETAILED DESCRIPTION

[0012] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0013] See also Figure 1-Figure 2The present invention provides a multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis, including a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module, a classification and identification module, and a model optimization module. Each module works together to realize automatic identification of copyright ownership.

[0014] The data acquisition module is composed of a multimodal data collector consisting of image, text, audio and video acquisition units, and is equipped with corresponding acquisition equipment. It collects, transmits and stores multimodal data according to the set process. The multimodal data preprocessing unit of the data preprocessing module can clean, convert the format, denoise and standardize the data to ensure data quality. The deep learning feature extractor of the feature extraction module uses CNN, RNN and MFS models to extract the features of images, videos, text and audio respectively.

[0015] The multimodal feature fusion network in the feature fusion module integrates multimodal features through feature-level fusion (such as splicing and weighted summation) and decision-level fusion (such as voting and weighting) to obtain the final classification result. The copyright ownership classifier of the classification and recognition module is based on a multi-task learning framework. After training, it predicts the copyright ownership of the fused features. The adaptive model optimizer of the model optimization module uses parameter updates, learning rate adjustment and regularization methods to improve model performance.

[0016] Multimodal data deep fusion technology utilizes data complementarity through feature-level and decision-level fusion to improve the accuracy of copyright information extraction. Specifically, it uses operations such as matrix splicing and voting. The automatic feature extraction algorithm automatically extracts key features of multi-dimensional data based on deep learning algorithms such as CNN, RNN, and MFS. CNN extracts image and video features through convolution pooling, RNN uses structures such as LSTM to capture text sequence features, and MFS calculates audio spectrum features. The multi-task learning framework collaboratively learns multiple tasks through a shared representation layer. In exclusive recognition, it uses a variety of similarity metrics and threshold setting methods to screen copyright information, thereby improving recognition efficiency and accuracy.

[0017] Data acquisition module: (1) Composition of the multimodal data collector: The multimodal data collector mainly consists of the following parts: image acquisition unit, text acquisition unit, audio acquisition unit, and video acquisition unit. Each unit is equipped with corresponding sensors and acquisition equipment to ensure the diversity and integrity of the data.

[0018] (2) Data collection process: The data collection process includes the following steps: Determine the collection objectives: Based on the needs of copyright identification, clarify the types of images, texts, audio and video to be collected.

[0019] Set collection strategies: Develop appropriate collection frequency, duration, and storage plans based on data characteristics.

[0020] Implement collection: Start the collection equipment and collect data according to the preset strategy.

[0021] Data transmission and storage: The collected data is transmitted to the data preprocessing module in real time and classified and stored.

[0022] Data preprocessing module: (1) Functions of the multimodal data preprocessing unit: The multimodal data preprocessing unit mainly includes operations such as data cleaning, format conversion, denoising and standardization.

[0023] (2) Data preprocessing process Data cleaning: Remove invalid, erroneous and duplicate data.

[0024] Format conversion: Convert data in different formats into a unified format for easy subsequent processing.

[0025] Denoising: Use filtering, smoothing and other techniques to remove noise from data.

[0026] Standardization: Normalize the data to make it have a uniform scale.

[0027] Feature extraction module: (1) Composition of deep learning feature extractor: Deep learning feature extractor includes models such as convolutional neural network (CNN), recurrent neural network (RNN) and Mel spectrum frequency spectrum (MFS).

[0028] (2) Feature extraction process: Image and video feature extraction: Use CNN to extract local features of images and videos.

[0029] Text feature extraction: Use RNN to capture the sequence features of text data.

[0030] Audio feature extraction: Extract the spectral features of audio data through MFS.

[0031] Feature fusion module: (1) Composition of multimodal feature fusion network: Multimodal feature fusion network mainly includes two levels: feature-level fusion and decision-level fusion.

[0032] (2) Feature fusion strategy: Feature-level fusion: concatenate and weighted sum feature vectors of different modalities.

[0033] Decision-level fusion: Voting, weighting and other strategies are performed on the classification results of each modality to obtain the final classification result.

[0034] Classification and recognition module: (1) Composition of the copyright attribution classifier: The copyright attribution classifier is based on a multi-task learning framework and includes multiple sub-networks, each responsible for learning different tasks.

[0035] (2) Classification and identification process Model training: Use the labeled dataset to train the classifier.

[0036] Classification prediction: The fused features are input into the classifier, and the copyright ownership prediction result is output.

[0037] Model optimization module: (1) Composition of the adaptive model optimizer: The adaptive model optimizer mainly includes parameter update strategy, learning rate adjustment, regularization and other methods.

[0038] (2) Model optimization process Parameter updating: Dynamically adjust model parameters based on new data.

[0039] Learning rate adjustment: Adjust the learning rate based on the performance of the model on the validation set.

[0040] Regularization: Use L1, L2 regularization and other methods to prevent model overfitting.

[0041] Multimodal data deep fusion technology: (1) Technical principle: The core principle of the feature-level fusion and decision-level fusion methods is to make full use of the complementarity of different modal data to improve the comprehensiveness and accuracy of copyright information extraction.

[0042] (2) Technical implementation Feature-level fusion: Image, text, audio, and video features are fused through methods such as matrix splicing and feature weighting.

[0043] Decision-level fusion: Using strategies such as voting and Bayesian reasoning to fuse the classification results of each modality.

[0044] Automatic feature extraction algorithm: (1) Technical principle: Deep learning algorithms (CNN, RNN, MFS) can automatically extract key features from multi-dimensional data, providing an accurate data basis for copyright identification.

[0045] (2) Technical implementation CNN: Extracts local features of images and videos through operations such as convolution and pooling.

[0046] RNN: uses structures such as long short-term memory (LSTM) to capture the sequential characteristics of text data.

[0047] MFS: Calculates the Mel spectrum of audio data and extracts spectral features.

[0048] Multi-task learning framework: Technical principle: The multi-task learning framework realizes collaborative learning of multiple related tasks by sharing the representation layer, thereby improving recognition efficiency and classification accuracy.

[0049] Similarity calculation: Similarity calculation is a key step in the exclusive recognition mechanism. We use a variety of similarity metrics, including cosine similarity, Euclidean distance, and Jaccard similarity coefficient, to adapt to different types of data characteristics.

[0050] Threshold setting: Through cross-validation and grid search techniques, we determined the optimal similarity threshold, which is crucial for excluding highly non-unique copyright information.

[0051] Exclusive screening: After similarity calculation, we consider features below a set threshold as highly unique copyright information and pass them on to the subsequent classification and recognition module. This process ensures the system's recognition results are highly reliable and accurate.

[0052] System performance evaluation and testing: Evaluation Metrics: To comprehensively evaluate the performance of our system, we used the following metrics: Accuracy: measures the proportion of copyright ownership correctly classified by the system.

[0053] Precision: measures the system's ability to correctly identify the positive class (copyright ownership).

[0054] Recall: measures the system’s ability to identify all positive classes.

[0055] F1 Score: The harmonic mean of precision and recall, which comprehensively reflects the performance of the system.

[0056] Runtime: Evaluates the processing speed and efficiency of the system.

[0057] Testing process: Data preparation: Collect test data from multiple data sources to ensure data diversity and representativeness.

[0058] System deployment: Deploy the system to the target environment and configure the necessary hardware and software resources.

[0059] Performance testing: Run the system, record various evaluation indicators, and compare them with expected performance.

[0060] Result analysis: Analyze the test results, identify the strengths and weaknesses of the system, and provide a basis for subsequent optimization.

[0061] 4. System optimization and future development direction: System optimization: In response to the problems found in the test, we optimized the system as follows: Adjust the feature extraction network structure to improve the representativeness of features.

[0062] Optimize feature fusion strategies to enhance the complementarity between different modal data.

[0063] Improve the exclusive identification algorithm to enhance the uniqueness and reliability of identification.

[0064] Future development direction: Data augmentation: Explore more effective data augmentation techniques to improve the model's adaptability to diverse data.

[0065] Model lightweighting: Research model compression and acceleration technologies to adapt to mobile devices and edge computing scenarios.

[0066] Combining artificial intelligence with law: Integrating legal expertise to improve the system's decision-making support capabilities in resolving copyright disputes.

[0067] Cross-domain application: Apply the methodology of this system to other fields, such as trademark recognition, patent analysis, etc.

[0068] Through the detailed system architecture, inventive concept description, performance evaluation, and optimization directions described above, the proposed multimodal intelligent analysis and multi-dimensional automatic identification system for copyright ownership, integrated with deep learning, demonstrates its innovation, practicality, and broad application prospects. The implementation of this system will not only improve the efficiency and accuracy of copyright protection, but also promote the development of copyright management-related technologies, providing strong technical support for intellectual property protection.

[0069] Example: Example 1: Detailed implementation Implementation of the data collection module: To ensure the efficiency and accuracy of data collection, we took the following detailed steps: Configuration and optimization of acquisition equipment Selection and customization of web crawlers: We have customized multiple Scrapy crawlers based on different website structures and content characteristics. Each crawler has specific crawling rules and exception handling mechanisms.

[0070] Selection and maintenance of data sources: We have established a dynamically updated data source library, which includes mainstream social media, news websites, online libraries, etc., to ensure the breadth and authority of the data.

[0071] Setting the data crawling frequency: We set different crawling cycles based on the activity and update frequency of the data source, ranging from hourly to weekly.

[0072] Expansion and processing of data types We have added support for multiple data formats, including but not limited to PNG, GIF, DOCX, WAV, etc., each of which has a dedicated parser for processing.

[0073] For data in special formats, we have developed customized parsing tools. For example, for GIF animations, we have implemented frame separation and keyframe extraction functions.

[0074] Implementation of the data preprocessing module: Data preprocessing is a key step in ensuring data quality. We performed the following operations: Refinement of the pre-processing process Image preprocessing: We use a series of image processing techniques, including histogram equalization, sharpening, color space conversion, etc., to improve the clarity and recognizability of the image.

[0075] Text preprocessing: We use natural language processing technology to conduct in-depth analysis of text, including sentiment analysis and topic modeling, to extract deeper semantic information.

[0076] Audio preprocessing: We extract rich music and speech features from audio data through technologies such as spectrum analysis and pitch tracking.

[0077] Video preprocessing: We have developed a video content analysis system that can automatically identify and extract key scenes and characters in videos.

[0078] Optimization and integration of implementation tools We have developed an integrated preprocessing platform that integrates APIs of multiple open source libraries and commercial software to achieve efficient automation of data preprocessing.

[0079] Implementation of the feature extraction module: Feature extraction is the core of copyright identification. We performed the following work: Selection and adjustment of feature extraction algorithms We compared various convolutional neural network (CNN) models, including VGG, ResNet, Inception, etc., and determined the optimal network structure and parameter settings through extensive experiments.

[0080] For text features, we not only used the BERT model, but also explored contextual embedding technologies such as ELMo and FLAIR to capture the deep semantics of text.

[0081] Implementation of the feature fusion module: The purpose of the feature fusion module is to effectively integrate features from different modalities. We adopted the following strategies: In-depth study of fusion strategies We studied multiple fusion techniques, including feature-level fusion, decision-level fusion, and model-level fusion, and determined the optimal fusion strategy through cross-validation experiments.

[0082] We also explore the use of attention mechanisms to dynamically adjust the importance of different features to improve the fusion effect.

[0083] Implementation of the Classification and Identification Module: The goal of the classification and identification module is to accurately classify data into the corresponding copyright ownership. We focus on the following aspects: Construction and training of classification models We designed a multi-layer perceptron (MLP) model and achieved high-precision classification performance by adjusting the network structure, activation function, and loss function.

[0084] We also adopted transfer learning techniques to leverage pre-trained model weights to accelerate the training process and improve the generalization ability of the model.

[0085] Implementation of the model optimization module: To improve the performance of the model, we implemented the following optimization measures: Exploration and application of optimization methods We compared various optimization algorithms, including SGD, Adam, RMSprop, etc., and found the optimization strategy that best suits the current task through hyperparameter tuning.

[0086] We also adopted techniques such as early stopping and model integration to prevent overfitting and improve model stability.

[0087] Example 2: Variant Implementation: In Example 2, we provide the following variant implementation to demonstrate the flexibility and diversity of the present invention: Variants of the data acquisition module We studied how to use IoT devices for data collection, including sensor networks, smart cameras, etc., and developed corresponding data synchronization and storage solutions.

[0088] We also explored the use of crowdsourcing to collect data and designed incentive mechanisms to encourage users to upload works with copyright information.

[0089] Variants of the feature extraction module We studied deep learning-based temporal feature extraction methods, such as LSTM and GRU, for processing time series information in audio and video data.

[0090] We also explored the use of graph neural networks (GNNs) to extract structured information from images, which is particularly important for identifying certain copyrighted works.

[0091] Variants of the feature fusion module We attempted to use deep reinforcement learning to achieve feature fusion and improve classification performance by learning the best fusion strategy.

[0092] We also studied how to combine traditional machine learning algorithms and deep learning models to achieve more powerful feature representation and classification capabilities.

[0093] Example 3: Extended Implementation In Example 3, we describe in detail how to further improve the performance and flexibility of the system. The following is a detailed description of the extended implementation of each module: Extension of data acquisition module Cross-platform data collection: By developing data collection tools suitable for different platforms, the system can seamlessly collect data from multiple sources, including but not limited to mobile applications, databases, and offline resources, ensuring the comprehensiveness and diversity of the data.

[0094] Artificial Intelligence Assistant: Using natural language processing technology, the AI ​​assistant can understand and extract copyright-related information, thereby improving the intelligence level of data collection.

[0095] Extension of data preprocessing module Big data processing framework: The introduction of big data processing frameworks such as Apache Spark can effectively process and analyze large-scale data sets and improve the efficiency of data preprocessing.

[0096] Intelligent denoising technology: Automatically identify and remove noise and irrelevant information in data through algorithms to ensure the quality of pre-processed data.

[0097] Extension of feature extraction module Multi-scale feature extraction: Through multi-scale analysis, the system can capture the features of data at different scales and improve the comprehensiveness of the features.

[0098] GAN-based feature extraction: Using generative adversarial networks, more representative features can be generated and the expressive power of features can be enhanced.

[0099] Extension of feature fusion module Multi-task learning framework: Implementing feature fusion and training of multiple related tasks in the same network can improve learning efficiency and model generalization ability.

[0100] Feature fusion strategy based on meta-learning: Through meta-learning technology, it can quickly adapt to different data distributions and task requirements, and improve the flexibility of feature fusion.

[0101] Extension of the classification and recognition module Multi-label classification method: allows a work to belong to multiple copyright categories at the same time, more accurately reflecting the copyright attributes of the work.

[0102] Federated learning technology: While protecting user privacy, it enables model training and classification across multiple institutions, enhancing the model's generalization capabilities and data security.

[0103] Extension of the Model Optimization Module Automated hyperparameter optimization: Use algorithms such as Bayesian optimization to automatically adjust model parameters to improve the efficiency and performance of model training.

[0104] Model interpretability technology: Through technologies such as LIME and SHAP, the interpretability of the model is enhanced and users' trust in the model is improved.

[0105] System integration and deployment Modular system architecture: Through modular design, each part of the system can be updated and maintained independently, improving the flexibility and maintainability of the system.

[0106] Cloud service and edge computing deployment: Depending on different computing requirements, the system can be deployed in cloud service or edge computing environments to ensure system stability and response speed.

[0107] Through these implementation methods, the system can operate efficiently in a wider range of scenarios, providing a more powerful and flexible solution for copyright ownership identification.

[0108] The embodiments of the present invention are presented for purposes of illustration and description and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments are chosen and described in order to better illustrate the principles of the invention and its practical application and to enable those skilled in the art to understand the invention and design various embodiments with various modifications as suited for specific applications.

Claims

1. A multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis, characterized by: It includes data acquisition module, data preprocessing module, feature extraction module, feature fusion module, classification and recognition module and model optimization module. Each module works together to realize automatic identification of copyright ownership.

2. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis as described in claim 1 is characterized by: The data acquisition module is composed of a multimodal data collector consisting of image, text, audio and video acquisition units, and is equipped with corresponding acquisition equipment to collect, transmit and store multimodal data according to the set process.

3. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis as described in claim 1 is characterized by: The multimodal data preprocessing unit of the data preprocessing module can clean, convert formats, denoise and standardize data to ensure data quality.

4. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis as claimed in claim 1 is characterized by: The deep learning feature extractor of the feature extraction module uses models such as CNN, RNN and MFS to extract features of images, videos, text and audio respectively.

5. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis as claimed in claim 1 is characterized by: The multimodal feature fusion network in the feature fusion module integrates multimodal features through feature-level fusion (such as splicing and weighted summation) and decision-level fusion (such as voting and weighting) to obtain the final classification result.

6. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis as claimed in claim 1 is characterized by: The copyright ownership classifier of the classification and recognition module is based on a multi-task learning framework and predicts copyright ownership based on fused features after training.

7. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis as claimed in claim 1 is characterized by: The Model Optimization module's adaptive model optimizer uses parameter updates, learning rate adjustments, and regularization methods to improve model performance.

8. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis as claimed in claim 1 is characterized by: Multimodal data deep fusion technology utilizes data complementarity through feature-level and decision-level fusion to improve the accuracy of copyright information extraction. Specifically, it uses operations such as matrix splicing and voting.

9. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multimodal intelligent analysis as claimed in claim 1 is characterized by: The automatic feature extraction algorithm automatically extracts key features of multi-dimensional data based on deep learning algorithms such as CNN, RNN, and MFS. CNN extracts image and video features through convolutional pooling, RNN uses structures such as LSTM to capture text sequence features, and MFS calculates audio spectrum features.

10. The multi-dimensional automatic identification technology for copyright ownership that integrates deep learning and multi-modal intelligent analysis as claimed in claim 1 is characterized by: The multi-task learning framework collaboratively learns multiple tasks through a shared representation layer, and uses multiple similarity metrics and threshold setting methods in exclusive identification to screen copyright information, thereby improving recognition efficiency and accuracy.

Images