An AI-powered intelligent photo and audio file management method and device

Abstract

This application discloses an AI-powered intelligent photo and audio-visual archive management method and apparatus, comprising the following steps: S1, acquiring user-uploaded photo archives, audio-visual archives, and artificial event tags, parsing them, associating the artificial event tags, and outputting original photo and audio-visual archive data with metadata; S2, extracting features from the original photo and audio-visual archive data output in S1 based on a preset AI model, and outputting feature sets for the photos and audio-visual archives; S3, constructing a spatiotemporal, entity, and event knowledge graph based on the feature sets output in S2, and outputting the spatiotemporal, entity, and event knowledge graph and event archive package; S4, parsing the user's search request to extract keywords, matching corresponding event nodes, recalling associated multimodal archives, and outputting them; This structure can achieve cross-modal association of photo and audio-visual archives by constructing a spatiotemporal semantic knowledge graph, solving the problems of event fragmentation and inefficient retrieval in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of archives management technology, specifically to an AI-powered intelligent photo and audio-visual archives management method and apparatus. Background Technology

[0002] In existing archival management technologies, photographic and audio-visual archives are mostly managed independently: photographic archives only achieve single-type classification, labeling, restoration, and copyright detection; audio-visual archives only achieve single-type content analysis, subtitle processing, and cataloging.

[0003] The above technology has the following drawbacks: 1. Lack of cross-modal association: Photo, audio, and video files are stored in a scattered manner, making it impossible to achieve cross-type association based on the same event, and making it difficult to reconstruct the complete event scene; 2. Low search efficiency: Users need to search for different types of files separately, and cannot retrieve related content with one click through event or semantic dimensions; 3. Insufficient structuring: The lack of a structured organization of the archives makes it difficult to uncover the event-related value of the archives.

[0004] Therefore, there is an urgent need to develop an AI-powered intelligent photo and audio file management method and device to solve the problems in the existing technology. Summary of the Invention

[0005] The purpose of this invention is to provide an AI-powered intelligent photo and audio-visual archive management method and device, which can realize cross-modal association of photos and audio-visual archives by constructing a spatiotemporal semantic knowledge graph, solving the problems of event fragmentation and inefficient retrieval in the prior art. Moreover, it has a simple structure and is easy to use, thus solving the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: An AI-powered intelligent photo and audio file management method includes the following steps: S1. Obtain user-uploaded photo files, audio files, and human event tags; parse the metadata of the photo files to extract the shooting time and location; parse the metadata of the audio files to extract the recording timestamp and GPS information; associate the human event tags; and output the original photo and audio file data with metadata. S2. Based on the preset AI model, extract features from the original photo and audio-visual archive data output by S1, and output the feature set of the photo's list of people, scene type, shooting time and shooting location, as well as the feature set of the audio-visual's speech-to-text, keyframe features and recording timestamp. S3. Based on the feature set output by S2, construct a spatiotemporal, entity, and event knowledge graph. The knowledge graph includes time nodes, spatial nodes, entity nodes, and event nodes. Establish the relationship between each node through preset rules, and then integrate the photos, audio, and video files under the same event node to generate an event file package. Output the spatiotemporal, entity, and event knowledge graph and the event file package. S4. Analyze the user's search request to extract keywords, match the corresponding event nodes based on the spatiotemporal, entity, and event knowledge graph, recall the associated multimodal files, sort them according to preset rules, generate a visualization display scheme, and output it.

[0007] By adopting the above technical solutions, we can complete the collection of archival data and extraction of metadata, the extraction of AI model features, and the construction of spatiotemporal, entity, and event knowledge graphs and the generation of event archive packages, ultimately achieving accurate retrieval and visualization of multimodal archives.

[0008] As a further aspect of the present invention: in S2, the preset AI model includes a photo feature extraction model and an audio-visual feature extraction model; the photo feature extraction model and the audio-visual feature extraction model are respectively used to extract the features of people and scenes in the photo and to extract the speech-to-text and keyframe features of the audio-visual recording.

[0009] By adopting the above technical solutions, the classification and core uses of the preset AI models in the multimodal feature extraction stage are clarified, ensuring that the feature extraction work of photos and audio-visual archives is accurately matched and carried out in an orderly manner, providing suitable feature data for subsequent knowledge graph construction.

[0010] As a further aspect of the present invention: the training process of the photo feature extraction model includes: acquiring a public portrait dataset and a custom archive portrait dataset, dividing them into a training set and a validation set according to a preset ratio, training them using a preset optimizer and a preset loss function, and stopping training when the accuracy of the validation set reaches a preset standard; the custom archive portrait dataset contains archive photos labeled with personal information and facial feature points.

[0011] By adopting the above technical solution, a complete training implementation method for the photo feature extraction model is provided, ensuring that the model has the ability to accurately extract the features of people and scenes in photos, laying the foundation for the effective association of photo archives.

[0012] As a further aspect of the present invention: the audio-visual feature extraction model includes a speech transcription model and a video keyframe feature extraction model; the speech transcription model, after being fine-tuned with archival terminology, transcribes the segmented audio segments; the video keyframe feature extraction model is used to extract the temporal features and keyframe image features of the video.

[0013] By adopting the above technical solution, the internal composition of the audio-visual feature extraction model and the processing method of each sub-model are clarified, ensuring that the speech-to-text and keyframe features of audio-visual archives can be accurately extracted, meeting the needs of cross-modal association.

[0014] As a further aspect of the present invention: in S3, the time node is constructed according to a preset format based on the photo shooting time and audio recording timestamp; the spatial node is constructed according to a preset format based on the GPS information of the photo and audio recording; the entity node includes people and scenes in the key frames of the photo and audio recording; and the event node is generated based on user tags and semantic extraction results.

[0015] By adopting the above technical solutions, the construction basis and format of each core node in the spatiotemporal, entity, and event knowledge graph are standardized, providing a unified and clear basic standard for establishing the relationship between each node.

[0016] As a further aspect of the present invention: the preset rules include: the rules for time nodes and space nodes include that one of them occurs in the other; the rules for entity nodes and event nodes include that one of them participates in or includes the other; the rules for photos, audio-visual files and event nodes include that one of them belongs to the other.

[0017] By adopting the above technical solutions, the association rules between nodes and archives in the knowledge graph are clarified, ensuring that different types of nodes and archives can be effectively associated according to a unified logic, and realizing event-level integration of multimodal archives.

[0018] As a further aspect of the present invention: in step S4, the keywords include event keywords, entity keywords, and modal keywords; the visualization display scheme includes a timeline display and a spatial map display, wherein the timeline is used to display the time flow of the event, and the spatial map is used to mark the event location.

[0019] By adopting the above technical solutions, the keyword types of search requests and the specific forms of visualization are refined, the accuracy of search matching and the intuitiveness of result display are optimized, and the user experience of archives is improved.

[0020] This application also discloses an AI multimodal archive association management device based on spatiotemporal semantic knowledge graph, including a data acquisition module, a multimodal feature extraction module, a spatiotemporal semantic association module and a retrieval and display module connected in sequence, applying the above-mentioned AI intelligent photo and audio-visual archive management method; The data acquisition module is used to acquire user-uploaded photo files, audio files, and optional human event tags, parse the metadata of the photo files and audio files to extract time and location related information, and output the original photo and audio file data with metadata after associating with human event tags. The multimodal feature extraction module is used to receive the output data of the data acquisition module, extract the features of people and scenes in the photos and the speech-to-text and key frame features of the audio and video through a preset AI model, and output the corresponding multimodal feature set. The spatiotemporal semantic association module is used to receive the output data of the multimodal feature extraction module, construct a spatiotemporal, entity, and event knowledge graph and establish the association relationship between each node, generate an event archive package and output the knowledge graph and the event archive package; The retrieval and display module is used to receive user retrieval requests, match and recall associated multimodal files based on the spatiotemporal, entity, and event knowledge graph, sort them, generate a visualization display scheme, and output it.

[0021] By adopting the above technical solution, a device architecture for implementing the AI ​​intelligent archive management method is provided. Through four core modules connected in sequence, the complete implementation of the management method is ensured, and the association management of multimodal archives is supported.

[0022] As a further aspect of the present invention: the data acquisition module includes an EXIF ​​parsing unit, an audio-visual metadata parsing unit, and a tag association unit; the EXIF ​​parsing unit is used to parse the EXIF ​​information of the photo archive to extract the shooting time and shooting location, the audio-visual metadata parsing unit is used to parse the metadata of the audio-visual archive to extract the recording timestamp and optional GPS information, and the tag association unit is used to associate manual event tags.

[0023] By adopting the above technical solution, the internal components and functions of each unit of the data acquisition module are clearly defined, ensuring that the original files uploaded by users can be effectively collected, parsed, and tagged, and output original file data that meets the requirements of subsequent processing.

[0024] As a further aspect of the present invention: the multimodal feature extraction module includes a photo feature extraction submodule and an audio-visual feature extraction submodule; the photo feature extraction submodule includes a preset person recognition model and a preset scene recognition model, and the audio-visual feature extraction submodule includes a preset speech transcription model and a preset video keyframe extraction model.

[0025] By adopting the above technical solutions, the internal architecture of the multimodal feature extraction module and the core models contained in each sub-module are clarified, ensuring that the feature extraction work of photos and audio-visual archives is carried out in an orderly manner according to the subdivided process, thereby improving the accuracy and efficiency of feature extraction.

[0026] Compared with the prior art, the beneficial effects of the present invention are: the present invention realizes the event-level cross-modal association and integration of photographs and audio-visual archives, solving the fragmentation problem of scattered storage of the two types of archives and difficulty in restoring complete event scenes in traditional management; at the same time, it supports multi-dimensional accurate retrieval and visualization display based on knowledge graphs, greatly improving the efficiency and ease of use of archive retrieval, and strengthening the mining of the event value of archives.

[0027] Other features and advantages of the present invention will be disclosed in detail in the following detailed description and accompanying drawings. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall structure of an AI-powered intelligent photo and audio-visual archive management method according to an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] In this embodiment of the invention, an AI-powered intelligent photo and audio-visual file management method and apparatus are described below. Figure 1 As shown, it includes a data acquisition module, a multimodal feature extraction module, a spatiotemporal semantic association module, and a retrieval and display module connected in sequence, and includes the following steps: S1. Obtain user-uploaded photo files, audio files, and human event tags; parse the metadata of the photo files to extract the shooting time and location; parse the metadata of the audio files to extract the recording timestamp and GPS information; associate the human event tags; and output the original photo and audio file data with metadata. S2. Based on the preset AI model, extract features from the original photo and audio-visual archive data output by S1, and output the feature set of the photo's list of people, scene type, shooting time and shooting location, as well as the feature set of the audio-visual's speech-to-text, keyframe features and recording timestamp. S3. Based on the feature set output by S2, construct a spatiotemporal, entity, and event knowledge graph. The knowledge graph includes time nodes, spatial nodes, entity nodes, and event nodes. Establish the relationship between each node through preset rules, and then integrate the photos, audio, and video files under the same event node to generate an event file package. Output the spatiotemporal, entity, and event knowledge graph and the event file package. S4. Analyze the user's search request to extract keywords, match the corresponding event nodes based on the spatiotemporal, entity, and event knowledge graph, recall the associated multimodal files, sort them according to preset rules, generate a visualization display scheme, and output it.

[0031] Module 1: Data Acquisition Module Input: User-uploaded photo files, audio / video files, and optional human-generated event tags; photo files include EXIF ​​information, and audio / video files include audio and video files; Processing logic: Parse the EXIF ​​information of the photo to obtain the shooting time and location, and convert the GPS coordinates into the format of province, city, and specific location; parse the metadata of the audio and video archive to obtain the timestamp and GPS information; associate it with the manual event tags entered by the user; Output: Original archive data of photos and audio-visual materials with metadata, including time, location, and event tags.

[0032] Module 2: Multimodal Feature Extraction Module This module extracts semantic features from photos and audio / video materials using an AI model. The model training process is as follows: 1) Photo Feature Extraction Submodule Function: Extract features of people and scenes from photos; Input: The original photo files output by the data acquisition module; AI Model and Training Process: Person recognition model: Training dataset: MS-Celeb-1M public portrait dataset + custom archive portrait dataset, the custom dataset contains 100,000 archive photos labeled with names and facial feature points; In one feasible embodiment, to ensure that no personal privacy information is involved in the training dataset, code names are used to replace names, and only the preset code names of individuals are collected instead of individual names.

[0033] Model structure: ResNet-50 backbone network and Transformer encoder. The ResNet-50 backbone network is used to extract low-level facial features, and the Transformer encoder is used to enhance the semantic expression of the features. Training process: Divide the dataset into training and validation sets in an 8:2 ratio; use the cross-entropy loss function, the formula is as follows: ,in For real labels, Predict probabilities for the model; use the AdamW optimizer with an initial learning rate of 0.001, decaying by 0.1 every 10 epochs; train for 50 epochs, and stop training when the person recognition accuracy on the validation set is ≥98%; Scene recognition model: Training dataset: Places365 public scene dataset + archive scene dataset. The archive scene dataset contains 50,000 archive photos labeled with "scene type", such as venues and outdoors. Model structure: ViT, Visual Transformer; Training process: Same as the person recognition model, training stops when the scene recognition accuracy on the validation set is ≥95%; Output: Includes a list of people in the photo, scene type, shooting time, and a set of features of the shooting location.

[0034] 2) Audio-visual feature extraction submodule Function: Extract speech text and keyframe features from audio and video recordings; Input: The original audio and video files output by the data acquisition module; AI Model and Training Process: Speech-to-text model: Training dataset: LibriSpeech public audio dataset + Archival professional audio dataset, the Archival professional audio dataset contains 20,000 annotated meeting / interview recordings; Model structure: The basic Whisper-small model is fine-tuned to address archival terminology, such as archival cataloging and archival group number. Training process: The audio is segmented into 10-second segments and input into the model to transcribe the text; the CTC connection temporal classification loss function is used; the SGD optimizer is used with a learning rate of 0.01 and a momentum of 0.9; fine-tuning is performed for 20 rounds, and training is stopped when the validation set word error rate WER ≤ 5%; Video keyframe feature extraction model: Training dataset: ActivityNet public video dataset + archive video dataset. The archive video dataset contains 10,000 archive videos labeled with "keyframe scenes / people". Model structure: 3D-CNN and ResNet-50, where 3D-CNN is used to extract temporal features from the video and ResNet-50 is used to extract keyframe image features; Training process: Same as the photo scene recognition model, training stops when the keyframe feature matching accuracy on the validation set is ≥90%; The output includes audio and video: Audio: A set of features including speech-to-text transcription and recording timestamps; Video: A set of features including keyframes of people and scenes, speech-to-text transcription, and recording timestamps.

[0035] Module 3: Spatiotemporal Semantic Relation Module This module is the core innovation module, and its function is to build a knowledge graph of spatiotemporal, entity, and event information to achieve cross-modal archive association.

[0036] Input: Photo features and audio-visual features output by the multimodal feature extraction module; Processing logic, including knowledge graph construction: Node construction: Time nodes: The format is YYYY-MM-DD HH:MM:SS, determined based on the photo shooting time and audio / video recording timestamp; Spatial nodes: The format is province-city-specific location, determined based on GPS information from photos and audio / video recordings; Entity nodes include people and scenes in the keyframes of the photo, such as Zhang and the venue; in one feasible embodiment, they include person codes, such as Xiao Hong, the venue, Qiu Tian, ​​the outdoors, Da Bai, and room A. Event nodes: generated based on user tags and semantic extraction results, such as the 2025 Construction Industry Conference; Edge construction rules: Time nodes and spatial nodes are linked by the logic of when they occur, for example, the connection between 09:00 on 2025-05-10 and Beijing - Beijing International Convention Center; Entity nodes and event nodes are linked by edges through participation and inclusion logic, such as the association between Zhang and the 2025 Construction Industry Conference; Photos, audio-visual files, and event nodes are linked by their respective logical connections; Event archive generation: Automatically integrates photos, audio, and video archives under the same event node, and labels them with event name, participants, time range, and location information; Output: Spatiotemporal, entity, and event knowledge graphs, and event archives.

[0037] Module 4: Search and Display Module Input: The user's search request, such as a photo or audio recording of Mr. Zhang's speech at the 2025 construction industry conference; Processing logic: Parse keywords related to events, entities, and modalities in the search request; match corresponding event nodes based on the knowledge graph to retrieve associated photos and audio-visual files; sort the files in chronological order to generate a visualization display scheme; Output: Search results: Related photos, audio clips, and video clips; Visualization methods: Timeline, showing the timeline of events; Spatial map, marking the locations of events.

[0038] Complete process example Data collection: User uploaded meeting photos, EXIF: 2025-05-10 09:00, Location: Beijing International Convention Center; Meeting recording, timestamp 09:00-11:00; The recording of Zhang's speech, timestamped 09:30-09:45, is tagged with the event tag "2025 Construction Industry Conference"; Multimodal feature extraction: Photo extraction: Person: Zhang; Scene: Conference venue; Video extraction: Keyframes: Opening ceremony; Speech transcription: The theme of this conference is: Development of the construction industry; Audio transcription: Zhang XX's speech content is the development trend of the construction industry; Spatiotemporal semantic association: Construct an event node: 2025 XX Industry Conference; associated time: 2025-05-10; space: Beijing International Convention Center; entity: Zhang, and bind the corresponding photos, videos, and audio recordings; Search and Display: When a user searches for "Zhang's speech at the 2025 Construction Industry Conference", the system retrieves related photos, audio clips, and video clips of Zhang's speech, and displays them in a timeline format.

[0039] This invention provides an AI-powered intelligent photo and audio-visual archive management method and device, which can realize cross-modal association of photos and audio-visual archives by constructing a spatiotemporal semantic knowledge graph, solving the problems of event fragmentation and inefficient retrieval in existing technologies, and has high reliability.

[0040] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0041] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An AI intelligent photo and audio-visual archive management method, characterized in that, The method comprises the following steps: S1, obtaining a photo archive, an audio-visual archive and a manual event label uploaded by a user, parsing metadata of the photo archive to extract a shooting time and a shooting location, parsing metadata of the audio-visual archive to extract a recording timestamp and GPS information, and outputting photo and audio-visual original archive data with metadata after associating the manual event label; S2, performing feature extraction on the photo and audio-visual original archive data output by S1 based on a preset AI model, outputting a feature set of a person list, a scene type, a shooting time and a shooting location of the photo, and a feature set of audio-visual speech transcription text, a key frame feature and a recording timestamp; S3, constructing a time-space, entity and event knowledge graph based on the feature set output by S2, the knowledge graph comprising a time node, a space node, an entity node and an event node, establishing an association relationship between the nodes through a preset rule, and generating an event archive package by integrating the photo, audio and video archives under the same event node, and outputting the time-space, entity and event knowledge graph and the event archive package; S4, parsing a search request of a user to extract a keyword, matching a corresponding event node based on the time-space, entity and event knowledge graph and recalling associated multi-modal archives, generating a visual display scheme and outputting the visual display scheme after sorting according to a preset rule.

2. The AI intelligent photo and audio-visual archive management method of claim 1, wherein, In S2, the preset AI model comprises a photo feature extraction model and an audio-visual feature extraction model; the photo feature extraction model and the audio-visual feature extraction model are respectively used to extract the person and scene features of the photo and to extract the audio-visual speech transcription text and key frame features.

3. The AI intelligent photo and audio-video archive management method of claim 2, wherein, The training process of the photo feature extraction model comprises: obtaining a public portrait data set and a custom archive portrait data set, dividing them into a training set and a validation set according to a preset ratio, training them using a preset optimizer and a preset loss function, and stopping training when the accuracy of the validation set reaches a preset standard; the custom archive portrait data set contains archive photos labeled with person information and facial feature points.

4. The AI intelligent photo and audio-visual archive management method of claim 2, wherein, The audio-visual feature extraction model comprises a speech transcription model and a video key frame feature extraction model; the speech transcription model is fine-tuned with archive professional terms to transcribe the segmented audio segments; the video key frame feature extraction model is used to extract the time sequence features and key frame image features of the video.

5. The AI intelligent photo and audio-visual archive management method of claim 1, wherein, In S3, the time node is constructed based on the photo shooting time and the audio-visual recording timestamp according to a preset format, the space node is constructed based on the GPS information of the photo and audio-visual according to a preset format, the entity node comprises the person and scene in the photo and audio-visual key frames, and the event node is generated based on the user label and the semantic extraction result.

6. The AI intelligent photo and audio-video archive management method of claim 5, wherein, The preset rules comprise: the rules of the time node and the space node include one occurring in the other; the rules of the entity node and the event node include one participating in or containing the other; and the rules of the photo, audio-visual archive and event node include one belonging to the other.

7. The AI intelligent photo and audio-video archive management method of claim 1, wherein, The keywords include event keywords, entity keywords, and modal keywords; the visual display scheme includes a timeline display and a spatial map display, the timeline is used to display the time flow of events, and the spatial map is used to mark the event location.

8. An AI multi-modal archive association management device based on a spatio-temporal semantic knowledge graph, characterized in that, The method comprises a data collection module, a multi-modal feature extraction module, a spatio-temporal semantic association module, and a retrieval and display module connected in sequence, and the method is applied to the AI intelligent photo and audio-visual archive management method. The data collection module is configured to obtain photo archives, audio-visual archives, and optional artificial event labels uploaded by a user, analyze metadata of the photo archives and the audio-visual archives to extract time and location related information, and output photo and audio-visual original archive data with metadata after associating the artificial event labels. The multi-modal feature extraction module is configured to receive output data of the data collection module, extract human and scene features of the photo, speech transcription text and key frame features of the audio-visual, and output corresponding multi-modal feature sets. The spatio-temporal semantic association module is configured to receive output data of the multi-modal feature extraction module, construct a spatio-temporal, entity, and event knowledge graph and establish an association relationship between nodes, generate an event archive package, and output the knowledge graph and the event archive package. The retrieval and display module is configured to receive a retrieval request of a user, match and recall associated multi-modal archives based on the spatio-temporal, entity, and event knowledge graph, generate a visual display scheme after sorting, and output the visual display scheme.

9. The AI intelligent photo and audio-video archive management device of claim 8, wherein, The data collection module comprises an EXIF analysis unit, an audio-visual metadata analysis unit, and a label association unit; the EXIF analysis unit is configured to analyze EXIF information of the photo archives to extract shooting time and shooting location, the audio-visual metadata analysis unit is configured to analyze metadata of the audio-visual archives to extract recording time stamps and optional GPS information, and the label association unit is configured to associate artificial event labels.

10. The AI intelligent photo and audio-visual archive management device of claim 8, wherein, The multi-modal feature extraction module comprises a photo feature extraction submodule and an audio-visual feature extraction submodule; the photo feature extraction submodule comprises a preset human recognition model and a preset scene recognition model, and the audio-visual feature extraction submodule comprises a preset speech transcription model and a preset video key frame extraction model.

Images