Photo archive sorting method and apparatus, electronic device, and storage medium

Abstract

The application discloses a photo archive arrangement method and device, electronic equipment and a storage medium, and belongs to the technical field of archive management. The method comprises the following steps: grouping a plurality of photos based on photo information to obtain at least two photo groups; extracting element information and high-level semantic features of the photos in each photo group, wherein the high-level semantic features comprise global embedding vectors and scene semantic labels; generating a photo abstract of each photo group according to the element information and the high-level semantic features; analyzing photo elements of each photo based on the photo abstract to obtain a photo title of each photo group; and arranging the plurality of photos according to the photo abstract and the photo title. The method can automatically, accurately and structurally complete photo archive arrangement and description generation.

Description

Technical Field

[0001] This application belongs to the field of archival management technology, and in particular relates to a method, apparatus, electronic device and storage medium for organizing photographic archives. Background Technology

[0002] With the popularization of digital technology, the number of photo archives accumulated by institutions and individuals has grown exponentially. Traditional photo archive management mainly relies on manual sorting, cataloging, and indexing. However, manually browsing, identifying, and inputting massive amounts of photo information is time-consuming and labor-intensive. Manual judgment of the time, place, and events of photos is prone to subjective errors and omissions. Many historical photos lack key metadata, such as time, place, and people, which greatly reduces their archival value. Furthermore, manually written titles and summaries have inconsistent styles, making standardized management difficult.

[0003] While existing technologies include independent image recognition, OCR, or EXIF ​​reading tools, there is a lack of complete solutions that can systematically integrate multi-source information and deeply understand the semantic content of photos to directly serve archival management operations.

[0004] Therefore, there is an urgent need for a technical solution that can automatically, accurately, and structurally complete the organization and description generation of photo archives. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, apparatus, electronic device, and storage medium for organizing photographic archives, which can automatically, accurately, and structurally complete the organization and description generation of photographic archives.

[0006] Firstly, this application provides a method for organizing photographic archives, the method comprising:

[0007] Based on the photo information, multiple photos are grouped into data groups to obtain at least two photo groups;

[0008] Extract feature information and high-level semantic features from the photos in each photo group. The high-level semantic features include global embedding vectors and scene semantic labels.

[0009] Based on the element information and the high-level semantic features, a photo summary is generated for each photo group;

[0010] Based on the photo summary, the photo elements of each photo are analyzed to obtain the photo title of each photo group;

[0011] The multiple photos are archived based on the photo summary and the photo title.

[0012] According to one embodiment of this application, the step of grouping multiple photos based on photo information to obtain at least two photo groups includes:

[0013] Parse the metadata fields in the EXIF ​​of the photos, analyze the temporal proximity of each photo, and obtain the first similarity data;

[0014] The location text in the photo is identified, and geospatial clustering is performed to obtain the second similarity data.

[0015] Facial features are extracted from the photos, and facial consistency clustering is performed on photos containing the same core group of people to obtain third similarity data.

[0016] Based on the scene type and time series identified in the photos, the event continuity is analyzed according to scene similarity and time continuity to obtain fourth similarity data;

[0017] The confidence level is assessed based on the first similarity data, the second similarity data, the third similarity data, and the fourth similarity data to obtain the photo group.

[0018] According to one embodiment of this application, the element information includes time elements, location elements, people elements, event elements, and photographer elements. Extracting element information from the photos in each photo group includes:

[0019] The time information of the photograph is read and normalized to obtain the time element;

[0020] Based on the GPS coordinates in the EXIF ​​of the photo, the location elements are extracted. In the case of missing GPS coordinates, the location elements are extracted by recognizing place names in road signs and banners using OCR and combining them with named entity recognition technology.

[0021] The human features are obtained by performing face detection and feature comparison on the photograph.

[0022] Based on the image content, text, time, location, and people in the photograph, the event elements are obtained through event reasoning.

[0023] Extract the photographer's element from the Artist field of the EXIF ​​data of the photograph, the photo watermark OCR, or the device serial number association library.

[0024] According to one embodiment of this application, the step of extracting feature information from photos in each photo group further includes:

[0025] Construct a photo relationship graph for the photos within the photo group;

[0026] In the event that the time, location, people, event, or photographer elements of the target photo are missing, the target photo is supplemented by supplementing its element information based on the photo relationship graph and by passing messages according to the element information of the neighboring photos of the target photo.

[0027] According to one embodiment of this application, the photo summary includes a structured summary and a natural language summary. Generating a photo summary for each photo group based on the element information and the high-level semantic features includes:

[0028] The element information is fused to obtain the ISL representation of the photo, and the overall confidence level is calculated for the ISL representation;

[0029] If the overall confidence level is higher than a preset threshold, the key fields in the ISL representation are filled into a predefined language template to generate the structured summary;

[0030] If the overall confidence level is lower than or equal to the preset threshold, the ISL representation is converted into a text prompt and jointly encoded with the global embedding vector and the scene semantic label. The input sequence is fed into the sequence model to generate the natural language summary.

[0031] According to one embodiment of this application, the step of analyzing the photo elements of each photo based on the photo summary to obtain the photo title of each photo group includes:

[0032] Extract summary elements from the photo summary, perform structured parsing on the summary elements, and obtain photo elements corresponding to the element information;

[0033] Generate a photo title for the photo based on the photo elements;

[0034] Common features were obtained by performing commonality analysis on the key fields of each photo in the photo group;

[0035] Based on the common features, the ISL representations of the photos are weighted and aggregated to generate the photo titles of the photo group.

[0036] According to one embodiment of this application, generating a photo title for the photo based on the photo elements includes:

[0037] In the visual pathway, the image pixels and cue words of the photo are input into a multimodal large model to perceive visual content from the image and generate the first title of the photo;

[0038] In the ISL pathway, the ISL representation of the photo elements is converted into text prompt words and input into the generation model to generate a second title for the photo;

[0039] Based on the overall confidence level of the photo, the first title and the second title are merged or filtered to obtain the photo title.

[0040] Secondly, this application provides a photo archive organizing device, the device comprising:

[0041] The first processing module is used to group multiple photos based on photo information to obtain at least two photo groups;

[0042] The second processing module is used to extract feature information and high-level semantic features from the photos in each photo group. The high-level semantic features include global embedding vectors and scene semantic labels.

[0043] The third processing module is used to generate a photo summary for each photo group based on the element information and the high-level semantic features.

[0044] The fourth processing module is used to analyze the photo elements of each photo based on the photo summary to obtain the photo title of each photo group;

[0045] The fifth processing module is used to organize the multiple photos into files based on the photo summary and the photo title.

[0046] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the photo file organization method described in the first aspect above.

[0047] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the photo archive organization method described in the first aspect above.

[0048] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the photo archive organization method as described in the first aspect.

[0049] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the photo file organization method as described in the first aspect above.

[0050] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0051] The photo archive organization method, apparatus, electronic device, and storage medium provided in this application have the following advantages over the prior art:

[0052] (1) By intelligently grouping photos in a multimodal manner and combining element information and advanced semantic features, structured summaries and accurate titles are generated, realizing the automated and semantic leap from raw images to standardized archives. It can automatically, accurately and structurally complete the organization and description generation of photo archives, improve the integrity of archive metadata, the consistency of event semantics, and the accuracy and readability of title summaries. At the same time, it supports efficient subsequent retrieval and human-computer collaborative proofreading, effectively meeting the technical needs of professional institutions for highly standardized, traceable and scalable digital photo archive management.

[0053] (2) ISL is used as an intermediate representation language for photo semantics to establish a standardized information bridge between image understanding and text generation, provide a semantic alignment intermediate layer, transform element information into a clear and controllable standardized prompt word structure, which can be aggregated to form a GroupMeta structure, enhance the ability to understand context, make the generation of abstracts and titles traceable to a unified structure, realize the standardization of archival semantics and machine interpretability, and also provide machine-parsable semantic representations that can be used for archiving, retrieval and generation tasks. Attached Figure Description

[0054] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0055] Figure 1 This is one of the flowcharts illustrating the photo file organization method provided in this application embodiment;

[0056] Figure 2 This is a second schematic flowchart of the photo file organization method provided in the embodiments of this application;

[0057] Figure 3 This is the third flowchart illustrating the photo file organization method provided in this application embodiment;

[0058] Figure 4 This is a schematic diagram of the structure of the photo file organizing device provided in the embodiments of this application;

[0059] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0061] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0062] The following description, in conjunction with the accompanying drawings, details the photo archive organization method, photo archive organization device, electronic device, and readable storage medium provided in this application through specific embodiments and application scenarios.

[0063] The photo archive organization method can be applied to a terminal, specifically executed by the hardware or software within the terminal.

[0064] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0065] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0066] The photo file organization method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the photo file organization method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The photo file organization method provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0067] like Figure 1 As shown, the method for organizing photo archives includes:

[0068] Step 110: Based on the photo information, group the multiple photos into data groups to obtain at least two photo groups;

[0069] The photo data includes metadata, image content, embedded text, and derived semantic information.

[0070] In step 110, the Exchangeable Image File Format (EXIF) metadata of each photo is parsed, and the DateTimeOriginal metadata field is extracted as a timestamp; a time proximity threshold is set, and photos with a time interval less than the threshold are initially clustered.

[0071] If the photo contains Global Positioning System (GPS) coordinates, a density-based spatial clustering algorithm (DBSCAN) is used to spatially cluster the geographic coordinates. For photos without GPS data, optical character recognition (OCR) is used to identify text such as road signs, banners, and store signs in the image. Named entity recognition (NER) is used to extract location keywords, and a geographic knowledge base is used for fuzzy matching to assist in spatial grouping.

[0072] The RetinaFace algorithm is used for face detection, and the ArcFace algorithm is used to extract facial feature vectors. The appearance of the same person in multiple photos is clustered, and photos containing the same core group of people are grouped together.

[0073] ResNet is used to classify images into scenes, and time series analysis is combined to identify photos with the same scene category and consecutive time as the same event segment.

[0074] Finally, a photo relationship graph is constructed, where nodes are photos and edge weights are calculated by weighted fusion of four-dimensional similarity (time, geography, face, and scene). The final photo group is then output using a graph partitioning algorithm based on spectral clustering.

[0075] Step 120: Extract feature information and high-level semantic features from the photos in each photo group. The high-level semantic features include global embedding vectors and scene semantic labels.

[0076] The information elements include dimensions such as time, location, people, events, and photographers.

[0077] Global embedding vectors are high-dimensional dense vectors obtained by deep encoding the visual content of the entire image. They can comprehensively represent the overall semantic information of the image and are used for cross-modal alignment, similarity calculation, and semantic generation.

[0078] Scene semantic labels are discrete category names output by scene classification models, used to describe the overall environment or context of a photograph.

[0079] In step 120, the EXIF ​​time is read first. If it is missing, it is parsed from the date text or file name recognized by OCR and uniformly converted to ISO 8601 format.

[0080] The system primarily uses EXIF ​​GPS data. When data is missing, place names are extracted using OCR+NER, and then converted into structured addresses using a reverse geocoding service.

[0081] By comparing the face recognition model with a pre-built database of important people, unknown faces are automatically clustered and users are advised to annotate them. After annotation, feedback is provided to update the database of people.

[0082] Input images, OCR text, time, location, and people into a Transformer-based multimodal event reasoning model, and output standardized event types;

[0083] The photographer can be read from the Artist field of EXIF ​​data, or inferred by correlating the watermark OCR and device serial number with a known photographer's device database.

[0084] In the process of high-level semantic feature extraction, the Contrastive Language-Image Pre-training (CLIP) model is used to extract the global embedding vector of each photo. The scene classification model is used to extract and output the scene semantic label and its confidence of each photo. The scene classification model can be built based on the multimodal fusion model.

[0085] Step 130: Generate a photo summary for each photo group based on the element information and the high-level semantic features;

[0086] Among them, the photo summary is a structured natural language description of the content of a single photo or a group of photos. It is generated by fusing element information and high-level semantic features and covers core archival information such as time, place, people, events and photographers.

[0087] In step 130, all dimensions and features of the feature information are fused to construct an Intermediate Semantic Language (ISL), which includes field values, confidence levels, source identifiers, and external knowledge base entity identity documents (IDs).

[0088] Based on the ISL and global embedding vector of each photo, the confidence scores of each element are weighted and averaged to calculate the overall confidence score of the ISL, and a structured summary is generated using a template-driven approach.

[0089] If the overall confidence score of ISL is less than the confidence threshold, the ISL is converted into a natural language prompt, which is then fed into the sequence model along with the image embedding to generate a fluent natural language photo summary.

[0090] The duplicates of all photo summaries within a photo group are removed and key information is merged to obtain a unified photo summary for the photo group.

[0091] Step 140: Based on the photo summary, analyze the photo elements of each photo to obtain the photo title of each photo group;

[0092] The photo elements are six structured metadata items extracted from the photo, including time, location, people, event, photographer, and title placeholder, represented in ISL format, with confidence level and source identifier.

[0093] In step 140, key fields such as time, location, people, events, and photographer are structurally parsed from the summary of each photo.

[0094] In the single title generation stage, a dual-path strategy is adopted. In the visual path, the original image is input into the BLIP-2 multimodal large model to generate the first title; in the ISL path, the ISL is converted into cue words and input into T5 to generate the second title.

[0095] Based on ISL confidence, the outputs of the two channels are dynamically fused to generate a single title of 15–20 characters.

[0096] During the group-level title generation stage, the key fields of all photos in the group are analyzed, common features are extracted, the comprehensive weight of each photo is calculated, the ISL is weighted and aggregated, and then input into the BART decoder to generate the group title.

[0097] Step 150: Organize the multiple photos into files based on the photo summary and the photo title.

[0098] In step 150, a multimodal archive data structure (ArchiveItem) with ISL as the core key is uniformly stored, including, for example, the original image Uniform Resource Identifier (URI), hash value, and embedding vector; the elements, confidence level, and source chain of the ISL; single / group level titles and summaries; version history and manual revision records;

[0099] Construct a relational or NoSQL database, establish a metadata table for photo groups, a table for people's information, an event classification table, etc., and link them through foreign keys; provide a web-based human proofreading interface, allowing users to modify any element, and the system automatically records feedback for incremental learning; support exporting to standard formats such as CSV, JSON, and EAD, and integrate with the file management system; utilize contrastive learning to map ISL text and images to the same vector space, supporting intelligent retrieval such as image-to-text search and event-based photo group retrieval.

[0100] According to the photo archive organization method provided in this application, photos are intelligently grouped in a multimodal manner, and structured summaries and accurate titles are generated by combining element information and advanced semantic features. This achieves an automated and semantic leap from raw images to standardized archives, and can automatically, accurately, and structurally complete photo archive organization and description generation. It improves the integrity of archive metadata, the consistency of event semantics, and the accuracy and readability of title summaries. At the same time, it supports efficient subsequent retrieval and human-computer collaborative proofreading, effectively meeting the technical needs of professional institutions for highly standardized, traceable, and scalable digital photo archive management.

[0101] In some embodiments, the step of grouping multiple photos based on photo information to obtain at least two photo groups includes:

[0102] Parse the metadata fields in the EXIF ​​of the photos, analyze the temporal proximity of each photo, and obtain the first similarity data;

[0103] The location text in the photo is identified, and geospatial clustering is performed to obtain the second similarity data.

[0104] Facial features are extracted from the photos, and facial consistency clustering is performed on photos containing the same core group of people to obtain third similarity data.

[0105] Based on the scene type and time series identified in the photos, the event continuity is analyzed according to scene similarity and time continuity to obtain fourth similarity data;

[0106] The confidence level is assessed based on the first similarity data, the second similarity data, the third similarity data, and the fourth similarity data to obtain the photo group.

[0107] During the temporal proximity analysis, the DateTimeOriginal metadata field in the EXIF ​​of the photos is parsed, and preliminary grouping is performed based on a preset time threshold.

[0108] During geospatial clustering, the DBSCAN clustering algorithm is used to group photos with similar geographical locations together based on GPS coordinates in the EXIF ​​data. For photos without GPS data, location text information recognized by OCR can be used to assist in the determination.

[0109] In the process of face consistency clustering, a face recognition model is used to extract facial features, and a clustering algorithm is used to identify groups of people that appear repeatedly. Photos containing the same core group of people are grouped together.

[0110] In the process of scene and event continuity analysis, image classification models are used to identify the scenes in photos, such as conference rooms, stadiums, and restaurants. Combined with time series data, photos with the same scenes and sequential occurrences in time are classified as belonging to the same event. Image classification models are built on platforms such as ResNet or ViT.

[0111] During the group confidence assessment process, a confidence score is generated for each group result. The matching degree of time, place, people, and scene is combined to obtain the photo group.

[0112] In particular, low-confidence groups will be prompted for manual review.

[0113] In some embodiments, the element information includes time elements, location elements, people elements, event elements, and photographer elements. Extracting element information from the photos in each photo group includes:

[0114] The time information of the photograph is read and normalized to obtain the time element;

[0115] Based on the GPS coordinates in the EXIF ​​of the photo, the location elements are extracted. In the case of missing GPS coordinates, the location elements are extracted by recognizing place names in road signs and banners using OCR and combining them with named entity recognition technology.

[0116] The human features are obtained by performing face detection and feature comparison on the photograph.

[0117] Based on the image content, text, time, location, and people in the photograph, the event elements are obtained through event reasoning.

[0118] Extract the photographer's element from the Artist field of the EXIF ​​data of the photograph, the photo watermark OCR, or the device serial number association library.

[0119] During the extraction of time elements, the priority order is: EXIF ​​time > OCR recognition of date text in banners, slogans, etc. in photos > time information in file names.

[0120] Time standardization is needed to convert time information from different sources into a unified standard format, such as ISO 8601.

[0121] In the process of extracting location elements, GPS coordinates in EXIF ​​data are preferred and converted into structured addresses such as province, city, district, and street through reverse geocoding service. When GPS is missing, textual landmarks such as road signs, store signs, and unit names on banners in the photo are identified, and location entities are extracted using named entity recognition technology.

[0122] In addition, scene reasoning can be performed on photos to identify scenes such as beaches and snow-capped mountains, which can serve as supplementary descriptions of locations.

[0123] In the process of extracting human features, the RetinaFace algorithm is used for face detection, and the ArcFace model is used for feature extraction and comparison. It supports the construction of an important human figure database. For unknown human figures, clustering and user annotation are performed. The annotated human figure information is fed back to the human figure database for updating, enabling the model to learn continuously.

[0124] In group photos, relationships between people can be inferred based on their positions, postures, and other information, thus identifying the main figures.

[0125] During the event element extraction process, image content, recognized text, time, location, people and other information are input into an event reasoning model for multimodal information fusion.

[0126] A Transformer-based multimodal pre-trained model is employed, which learns the association between different modal information and event types through an attention mechanism to obtain an event reasoning model. For example, if the text "cake," "many people," "smiley face," and "happy birthday" are recognized, and the time is evening and the location is indoors, then the event can be inferred to be a "birthday party." The multimodal pre-trained model can be a variant of CLIP.

[0127] In addition, the system has a built-in extensible event classification system that generates event tags to output standardized event types. For example, event tags can be work meetings, cultural and sports activities, foreign affairs receptions, celebrations, etc.

[0128] During the process of extracting photographer elements, the information can be read from the Artist or Copyright field of EXIF, or identified from the edges of the photo, watermarks, or specific markings through OCR recognition; or the camera model serial number in EXIF ​​can be associated with a known database of photographer equipment.

[0129] In some embodiments, the extraction of feature information from photos in each photo group further includes:

[0130] Construct a photo relationship graph for the photos within the photo group;

[0131] In the event that the time, location, people, event, or photographer elements of the target photo are missing, the target photo is supplemented by supplementing its element information based on the photo relationship graph and by passing messages according to the element information of the neighboring photos of the target photo.

[0132] In practice, photos within the same group are treated as a graph structure, with nodes representing photos and edges representing the strength of association between photos based on factors such as temporal proximity, co-occurrence of people, and similarity of scenes.

[0133] A graph neural network is used to perform message passing and node representation learning on a photo relationship graph. For missing features in a photo, prediction and completion are performed by aggregating the corresponding feature information from its neighboring photos.

[0134] Construct an undirected weighted graph G=(V,E) for the same group of photos. Each photo is a node. The edge weights are determined by linear or non-linear combinations of similarity factors such as "same person / same place / same scene / temporal proximity":

[0135]

[0136] in, The weight of the overall similarity between photo i and photo j; , , , These are the weight coefficients for the four types of similarity, which can be set through training or experience to adjust the importance of time, location, face, and scene in the final weight. This is a temporal similarity index between photo i and photo j, reflecting the proximity of the two photos' shooting times. This is a geographic similarity index that reflects the degree of proximity between two locations (identified by GPS or OCR). The facial similarity index between photo i and photo j reflects the degree of similarity between the groups of people in the two photos; This is a scene similarity index between photo i and photo j, reflecting the degree of similarity in the overall scene categories of the two photos.

[0137]

[0138]

[0139]

[0140]

[0141] in, and The timestamps (time stamps) for photos i and j; The attenuation coefficient controls the sensitivity of the time difference to the similarity. The spatial distance between two GPS coordinates; and These are the GPS coordinates of photos i and j, respectively; Scale parameters for controlling the rate of decay due to location differences; and These are the facial feature embedding vectors for photos i and j, respectively; Cosine similarity is used to measure the consistency of a group of characters. and These are the scene semantic vectors for photos i and j, respectively, which can be output by the embedding of the scene classification model; Cosine similarity between scene features.

[0142] Use GraphSAGE for message passing to obtain node representations. For missing elements such as locations and events, prediction is made using classification / sequence labeling heads:

[0143]

[0144] in, This represents the feature representation of node i in the l-th layer GNN; Aggregation functions for GraphSAGE, such as mean / max / LSTM aggregation; In the graph, photos i and j are connected by an edge, meaning they have similarity. This represents the total number of layers in the GNN. The result is the prediction of missing features for photo i; This serves as a task header, used for classification (such as people / locations / events) or sequence labeling.

[0145] Evidence from multiple sources, including EXIF, OCR, image scene, face co-occurrence, and neighbor node prediction, is fused using Dempster–Shafer or weighted logit to output the final completion value and confidence level.

[0146]

[0147] in, This means that the prediction results from multiple sources are transferred into the logit space and then weighted and averaged to obtain the final logit prediction after fusion; and the confidence level is used to summarize the evidence from multiple sources, which is normalized by Sigmoid to obtain the final confidence level C. The final prediction result is obtained after multi-source fusion, such as the prediction of elements like location, event, and facial features; This represents the prediction result for the element from the k-th source. To convert the predicted probabilities into logit form, which facilitates linear fusion; The credibility weight of the k-th source is given, for example, EXIF ​​> OCR > inference; Number the information source, such as EXIF, OCR, image model inference, GNN completion, etc.; The final output is the overall confidence level, which represents the reliability of the prediction result and takes a value of 0 to 1. This is the Sigmoid function, used to map the weighted sum to the (0,1) interval; The original confidence score for the k-th source is such as EXIF ​​reliability, OCR text quality score, GNN inference confidence, etc. Let be the credibility weight of the k-th source, and represent the source quality weight, as in the previous formula. This is the information source number, consistent with the previous formula.

[0148] Using ISL as the core key, multimodal alignment and versioning are performed, retrieval and consistency space are established, and contrastive learning is used to map ISL text and images to the same vector space.

[0149]

[0150] in, The global embedding vector of the i-th photo is extracted by a visual encoder (such as ViT or CLIP image encoder) and is used to represent the overall visual semantic features of the photo. The ISL text embedding vector corresponding to the i-th photo is obtained by encoding the ISL structured semantic representation by a text encoder (such as a BERT / T5 encoder) and is used to represent the structured semantic information of the photo. The j-th candidate ISL text embedding is used as a negative sample in contrastive learning (if j=i, it is a positive sample). Image embedding vector With text embedding vectors The cosine similarity between visual and semantic features is used to measure the degree of consistency between them. The temperature coefficient is a positive real number used to adjust the "smoothness" of the softmax distribution. The lower the temperature, the more the difference in similarity is amplified.

[0151] The cross-modal alignment loss function (Contrastive Loss) between image embeddings and ISL text embeddings aims to improve the similarity of correct image-text pairs and reduce the similarity of incorrect matching pairs. It maps images and ISL text to a unified semantic space, enabling cross-modal retrieval capabilities such as image-to-text search and text-to-image search, and allowing for text-to-image, image-to-text, and ISL-based document search. Strongly consistent fields (time / location / people / event) employ optimistic locking and an approval queue; manual revisions generate new revisions. Each completed value stores "source=exif / ocr / infer" and "confidence," ensuring traceability and meeting archival auditing requirements.

[0152] In this embodiment, a multimodal archive data model with ISL as the core key integrates images, elements, generated text, source tracing, and version history into a unified structure; and supports fusion retrieval and traceable revision through a cross-modal common vector space.

[0153] In some embodiments, the photo summary includes a structured summary and a natural language summary, wherein generating a photo summary for each photo group based on the feature information and the high-level semantic features includes:

[0154] The element information is fused to obtain the ISL representation of the photo, and the overall confidence level is calculated for the ISL representation;

[0155] If the overall confidence level is higher than a preset threshold, the key fields in the ISL representation are filled into a predefined language template to generate the structured summary;

[0156] If the overall confidence level is lower than or equal to the preset threshold, the ISL representation is converted into a text prompt and jointly encoded with the global embedding vector and the scene semantic label. The input sequence is fed into the sequence model to generate the natural language summary.

[0157] Among them, ISL represents structured data, which has the characteristics of strong structural stability, can be adapted to templated archiving systems, has high clarity of prompts, reduces generation errors, and allows different archiving institutions to customize style templates based on the ISL representation. It can support style customization and cross-modal normalized expression.

[0158] To improve the accuracy and consistency of photo caption generation and metadata extraction, this application introduces ISL representation as an intermediate representation language for photo semantics. It is used to establish a standardized information bridge between image understanding and text generation, providing a semantically aligned intermediate layer that transforms element information into a clear and controllable standardized cue word structure. This structure can be aggregated to form a GroupMeta structure, enhancing contextual understanding capabilities. It ensures that both summary and caption generation are traceable to a unified structure, achieving archival semantic standardization and machine interpretability. It also provides a machine-parseable semantic representation that can be used for archiving, retrieval, and generation tasks.

[0159] For entities such as people and locations in ISL representations, entity linking technology is used to associate the ISL representations with entity IDs in external knowledge bases such as Wikidata and GeoNames, achieving cross-modal semantic alignment. For example, the location "park" not only stores text but also associates its geographic entity ID, laying the foundation for subsequent spatial reasoning.

[0160] ISL represents the text value, confidence level, source identifier, and external knowledge base entity ID of each element, and achieves semantic alignment between image features and ISL text embeddings through cross-modal contrastive learning.

[0161] For photos with clear rules and complete elements, predefined templates can be used to quickly generate summaries. For example, a template can be generated for "[Time], at [Location], [People] participated in [Event], taken by [Photographer]".

[0162] Employing a sequence-to-sequence model that uses multimodal features as input, this method generates more natural and flexible text summaries. Trained on "photo feature-summary" paired data, the model learns how to organize language.

[0163] The sequence-to-sequence model is constructed based on the T5 model or the Bidirectional and Auto-Regressive Transformers (BART) model.

[0164] Construct style condition vectors for organization s Among them, style condition vector The prototype obtained from the example title / summary by the text encoder, or represented by the parameter subspace of LoRA / Adapter, learns the organizational genre through a few-shot style sample library, injects style vectors into the title / summary generation network with an adapter structure, and jointly optimizes length, terminology and template constraints with a multi-objective function, thereby transferring and maintaining a consistent output style across different organizations.

[0165] Insert the Adapter / LoRA into the BART decoder:

[0166]

[0167] in, The hidden state vector of the original decoder represents the semantic representation of the content of the model without style injection; This is a new hidden state after style information is injected, used to generate titles or summaries that have consistency in organizational style and format. W1 is the low-rank trainable weight matrix of Adapter / LoRA, where W1 compresses the original hidden state hhh into a low-dimensional subspace (bottleneck layer), and W2 remaps the activated bottleneck representation back to the original dimension; U is the style injection matrix, which projects the institutional style vector zsz_szs onto the semantic space consistent with the bottleneck layer, and is used to inject style information into the model's decoding process. This is the style embedding of the organization, which is obtained by a text encoder from the example title / abstract of the organization. It is used to represent the language style and writing style of a specific organization (such as school style, corporate style, etc.). (.) represents a non-linear activation function (such as ReLU, SiLU, GELU), used to enhance expressive power and improve the model's ability to distinguish style condition changes; This is an intermediate representation resulting from the fusion of content semantics (from hhh) and "institutional style semantics" (from zsz_szs); To map the stylized intermediate representation back to the original latent state dimension, a residual structure is added to hhh to achieve low-cost style injection.

[0168] U injects style vectors into the bottleneck layer, with fine-tuning only updating... , , .

[0169] Multi-task optimization during training:

[0170]

[0171] This loss function achieves multi-objective joint optimization of "semantic correctness + length consistency + terminology consistency + style consistency", ensuring that the generated titles / abstracts are accurate, standardized, and conform to the unique writing style of the institution.

[0172] in, The final joint loss generated for the title / summary is used for end-to-end optimization; Cross-entropy loss constrains the semantic consistency between the model's generated title / summary and the target title / summary; it is the main loss term. Penalty for length constraint deviation; It is the hit rate of the organization's glossary; This results in a loss of consistency. These are the weighting coefficients for each auxiliary loss, used to adjust the importance of length, terminology, and style in the overall optimization.

[0173] During inference, specify the organization s, load the corresponding Adapter and terminology, and generate it based on ISL conditions.

[0174] Temperature and penalty terms are introduced to limit the introduction of new words and ensure consistency. A TextCNN / SVM style discriminator is used to guarantee style consistency.

[0175] For example, for the same ISL, time = 2025-04-12, location = park, event = spring outing, unit = Beijing Chaoyang No.1 Middle School.

[0176] The corporate style output is: images of the social practice activities of Chaoyang No.1 Middle School in the spring of 2025.

[0177] The school's style output is: 2025 Spring Outing - Group Photo in the Park.

[0178] In some embodiments, the step of analyzing the photo elements of each photo based on the photo summary to obtain the photo title of each photo group includes:

[0179] Extract summary elements from the photo summary, perform structured parsing on the summary elements, and obtain photo elements corresponding to the element information;

[0180] Generate a photo title for the photo based on the photo elements;

[0181] Common features were obtained by performing commonality analysis on the key fields of each photo in the photo group;

[0182] Based on the common features, the ISL representations of the photos are weighted and aggregated to generate the photo titles of the photo group.

[0183] For a single photo, extract the most essential elements from the abstract, usually following a concise format of event-location-time. Use models or rules to ensure the title length is within a limited range for easy browsing and retrieval, thus obtaining the photo title.

[0184] For a photo group, analyze the common elements of all photos in the group, such as common people, continuous events, the same location, etc., and extract the commonalities within the group. The photo titles of the photo group should be able to summarize the core content of the entire group, such as a photo collection of activities from start time to end time, unit name, and event name.

[0185] In actual implementation, the overall weight of each photo i within the group is... It is determined by time centrality, event representativeness, key figures, and visual quality:

[0186]

[0187] in, It is time centrality; It is representative of the event; It's about the core of the character; It's about visual quality; For weights.

[0188] Generate photo titles for photo groups for:

[0189]

[0190] in, The title for the generated photo group is used to indicate the core event, location, people, and time of the entire photo group; The ISL structured semantic representation of the i-th photo includes elements such as time, location, people, and events; (.) is an ISL text encoder that converts structured ISL semantics into fixed-dimensional text embedding vectors for subsequent aggregation and decoding; The weight of the i-th photo in the group-level title generation reflects the importance of the photo and is usually obtained by weighting factors such as time proximity, event representativeness, key person, and visual quality. The group-level semantic representation obtained by weighted summation represents the comprehensive semantic information of the entire group of photos; (.) is a text generation decoder that generates the final natural language title based on the aggregated group-level text vectors.

[0191] For example, if 5 photos are grouped together, spanning 2 hours, and based on the same core person and similar events, the output group title could be "Photo Collection of Spring Outing Activities in 2025".

[0192] Based on the confidence level of the photographic elements Dynamic selection or weighted fusion:

[0193]

[0194] in, This is the final output group-level heading; For titles generated by visual pathways (multimodal models, such as BLIP-2), the integrity and naturalness of the visual content should be emphasized; For titles generated via the ISL pathway, emphasis is placed on structure, standardization, and consistency with archival standards; ISL represents the overall confidence level, reflecting the reliability of structured elements such as time, place, people, and events; The low confidence threshold is set below CISLC_{\text{ISL}}CISL. When the value is below this threshold, the ISL information is considered unreliable and the visual title should be the primary indicator. This is a high confidence threshold. When CISLC_{\text{ISL}}CISL is higher than this value, the header can be output entirely based on ISL. This is the fusion coefficient, used for weighted fusion between the visual title and the ISL title. The formula is:

[0195]

[0196] in, A linear scaling factor between 0 and 1, used to dynamically adjust the blending ratio between the visual caption and the ISL caption based on ISL confidence level; when = When λ=0, it indicates that visual titles are used exclusively; when = When λ=1, it indicates that the ISL header is used in its entirety; when Located in the interval [ , When the interval is between ], linear interpolation is performed according to the proportion of that interval; , The upper and lower bounds of the confidence interval are used to determine the transition interval from "visual-based" to "structure-based".

[0197] For example, in =0.85, =0.6, With a value of 0.9 and λ=0.625, a fused title is output. Among the contribution ratios of the two title generation pathways, the structural pathway accounts for 62.5% and the visual pathway accounts for 37.5%, achieving a dynamic adaptive balance between generation stability and semantic consistency.

[0198] In this embodiment, by combining the attention mechanism with semantic focus, the group-level title generation of the event center frame is automatically focused. The ISL representation, summary and weight of all photos in the photo group are input into a sequence-to-sequence model. The sequence-to-sequence model naturally learns through the attention mechanism that the information of photos with high weight should be given more attention, and automatically and accurately generates a title that can summarize the entire group.

[0199] In some embodiments, generating a photo title for the photo based on the photo elements includes:

[0200] In the visual pathway, the image pixels and cue words of the photo are input into a multimodal large model to perceive visual content from the image and generate the first title of the photo;

[0201] In the ISL pathway, the ISL representation of the photo elements is converted into text prompt words and input into the generation model to generate a second title for the photo;

[0202] Based on the overall confidence level of the photo, the first title and the second title are merged or filtered to obtain the photo title.

[0203] Among them, the visual pathway is an end-to-end path that directly uses the original image to generate the title, without relying on structured semantics, and perceives the content from pixels through a multimodal large model.

[0204] Image pixels are RGB three-dimensional arrays obtained after decoding a digital photograph, serving as the raw input data for the visual pathway.

[0205] Prompts are natural language instructions used to guide multimodal large models in generating text that conforms to the expected format.

[0206] The multimodal large model is a pre-trained deep learning model that can jointly process images and text and has the ability to understand and generate images and text. It is built on BLIP-2.

[0207] The ISL pathway is a controllable path for generating titles based on Structured Intermediate Semantic Language (ISL), emphasizing element integrity and archiving standardization.

[0208] The ISL representation of photo elements is a unified expression of elements such as time, place, people, event, photographer, and title in a structured JSON format with confidence level, source, and normalized values.

[0209] Text prompts are natural language sentences automatically converted from ISL and used as input prompts for the plain text generation model.

[0210] The generative model is a sequence model for text-to-text generation, which receives structured prompts and outputs titles that conform to grammar and style.

[0211] The overall confidence score is a weighted or aggregated confidence score of each element in a single photo's ISL, used to determine fusion strategies.

[0212] In the actual execution, the photo to be processed is decoded into a pixel tensor in RGB format and resized to 224×224. At the same time, a text prompt is constructed, and the image tensor and the prompt are input together into a pre-trained multimodal large model. The image is converted into a visual token by the ViT encoder, and the prompt is encoded into a text token by the language model tokenizer. The two are fused in the cross-modal attention layer. The model generates a natural language sequence based on joint representation autoregression, and after removing redundant punctuation and truncating to within 20 characters, the first title is output.

[0213] The structured elements of the photograph are read from the constructed ISL representation, including time, location, key figures, event type, and photographer. Based on predefined template rules, the fields of the structured elements are populated to generate text prompts in natural language form. The prompts are then input into the T5-base plain text generation model, which generates a second title that conforms to the archive style based solely on the text context.

[0214] The overall confidence level of the photo's ISL is calculated by weighted averaging of the confidence levels of each element. Based on the overall confidence level, either the first or second title is determined as the photo title.

[0215] In this embodiment, an end-to-end intelligent processing flow is constructed based on the synergy and integration of image recognition, natural language processing, OCR recognition, and EXIF ​​data parsing. This flow simulates the organization and cataloging thinking of archival experts and enables intelligent summarization, title generation, and structured extraction of archival metadata for photo groups and individual photos. This achieves comprehensive intelligent management of photo archives from physical grouping to semantic description, improving the efficiency and accuracy of photo archive organization and filing.

[0216] The following is a specific example.

[0217] like Figure 2 As shown, in the input layer, photos are retrieved from the original photo library and EXIF ​​metadata is simultaneously fused with information from external knowledge bases (such as face databases and location databases).

[0218] In the core processing layer, unstructured images are transformed into standardized ISL representations through image parsing, intelligent grouping, six-element extraction, and ISL generator, thereby driving the two branches of summary generation and title generation respectively.

[0219] At the output layer, the generated structured metadata and photo / group titles are aggregated into the archive database, and a human-computer verification interface is provided to support manual review and feedback.

[0220] like Figure 3As shown, in the user interaction layer, users can upload photos and perform queries; in the image understanding module, object detection, face recognition, and scene analysis are performed to extract the original visual semantics; in the intermediate semantic language (ISL) layer, information is transformed into a structured description containing six elements: time, location, people, and events, and semantic anchoring is performed; in the context reasoning and information completion module, a graph neural network (GNN) is used to construct a photo relationship graph, and multi-source confidence is fused to intelligently complete missing elements; in the title generation module, a dual-path strategy of visual pathway and ISL prompt pathway is adopted to adaptively fuse and generate single photo titles; in the group-level title fusion module, a title summarizing the entire photo group is generated based on dynamic weight aggregation and event center extraction; in the archival style modeling module, lightweight fine-tuning techniques such as Adapter or LoRA are used to uniformly output text styles that conform to archival standards; all structured data is organized by the multimodal archival storage module with ISL as the primary key, and cross-modal indexes are constructed by combining contrastive learning to achieve efficient version management, long-term storage, and semantic-level intelligent retrieval, forming an end-to-end intelligent photo archive organization closed loop.

[0221] The photo file organization method provided in this application can be executed by a photo file organization device. This application uses a photo file organization device executing the photo file organization method as an example to illustrate the photo file organization device provided in this application.

[0222] This application also provides a photo archive organization device.

[0223] like Figure 4 As shown, the photo archive organization device includes:

[0224] The first processing module 410 is used to group multiple photos based on photo information to obtain at least two photo groups;

[0225] The second processing module 420 is used to extract feature information and high-level semantic features from the photos in each photo group. The high-level semantic features include global embedding vectors and scene semantic labels.

[0226] The third processing module 430 is used to generate a photo summary for each photo group based on the element information and the high-level semantic features.

[0227] The fourth processing module 440 is used to analyze the photo elements of each photo based on the photo summary to obtain the photo title of each photo group;

[0228] The fifth processing module 450 is used to organize the multiple photos into files based on the photo summary and the photo title.

[0229] The photo archive organization device provided in this application embodiment intelligently groups photos in a multimodal manner and combines element information and advanced semantic features to generate structured summaries and accurate titles. This achieves an automated and semantic leap from raw images to standardized archives, enabling automatic, accurate, and structured photo archive organization and description generation. It improves the integrity of archive metadata, the consistency of event semantics, and the accuracy and readability of title summaries. At the same time, it supports efficient subsequent retrieval and human-computer collaborative proofreading, effectively meeting the technical needs of professional institutions for highly standardized, traceable, and scalable digital photo archive management.

[0230] The photo file organization device in this application embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM or self-service machine, etc. The embodiments of this application do not specifically limit it.

[0231] The photo file organization device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0232] The photo file organizing device provided in this application embodiment can realize the various processes implemented in the photo file organizing method embodiment as described above. To avoid repetition, it will not be described again here.

[0233] In some embodiments, such as Figure 5As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored on the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-described photo file organization method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0234] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0235] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described photo archive organization method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0236] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0237] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described photo file organization method.

[0238] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0239] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described photo file organization method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0240] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0241] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0242] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the photo file organization method of the various embodiments of this application.

[0243] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0244] In the description of this application, "multiple" means two or more.

[0245] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0246] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0247] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for organizing photo archives, characterized in that... ,include: Based on the photo information, multiple photos are grouped into data groups to obtain at least two photo groups; For each photo group, extract feature information and high-level semantic features. The feature information includes time elements, location elements, people elements, event elements, and photographer elements. The high-level semantic features include global embedding vectors and scene semantic labels. Based on the element information and the high-level semantic features, a photo summary is generated for each photo group; Based on the photo abstracts, the photo elements of each photo are analyzed to obtain the photo titles for each photo group; The multiple photos are archived based on the photo abstracts and photo titles; The photo summary includes a structured summary and a natural language summary. The process of generating a photo summary for each photo group based on the element information and the high-level semantic features includes: The element information is fused to obtain the ISL representation of the photograph, and a comprehensive confidence score is calculated for the ISL representation. If the overall confidence level is higher than a preset threshold, the key fields in the ISL representation are filled into a predefined language template to generate the structured summary; If the overall confidence level is lower than or equal to the preset threshold, the ISL representation is converted into a text prompt, and jointly encoded with the global embedding vector and the scene semantic label. The input sequence is fed into the sequence model to generate the natural language summary. Based on the photo summary, the photo elements of each photo are analyzed to obtain the photo title of each photo group, including: Extract summary elements from the photo summary, perform structured parsing on the summary elements, and obtain the photo elements corresponding to the element information; Generate a photo title for the photo based on the photo elements; Common features were obtained by performing commonalities analysis on key fields of each photo in the photo group; Based on the common features, the ISL representations of the photos are weighted and aggregated to generate photo titles for the photo group; The step of generating a photo title based on the photo elements includes: In the visual pathway, the image pixels and cue words of the photograph are input into a multimodal large model to perceive visual content from the image and generate the first title of the photograph; In the ISL pathway, the ISL representation of the photo elements is converted into text prompt words and input into the generation model to generate a second title for the photo; Based on the overall confidence level of the photo, the first title and the second title are merged or filtered to obtain the photo title.

2. The photo archive organization method according to claim 1, characterized in that... The process involves grouping multiple photos based on photo information to obtain at least two photo groups, including: The metadata fields in the EXIF ​​data of the photos are parsed, and the temporal proximity of each photo is analyzed to obtain the first similarity data; The location text in the photo is identified, and geospatial clustering is performed to obtain second similarity data; Facial features are extracted from the photos, and photos containing the same core group of people are clustered based on facial consistency to obtain third similarity data; Based on the scene type and time series identified in the photos, and by analyzing the event continuity according to scene similarity and time continuity, fourth similarity data is obtained. The confidence level is assessed based on the first similarity data, the second similarity data, the third similarity data, and the fourth similarity data to obtain the photo group.

3. The photo archive organization method according to claim 1, characterized in that... The step of extracting feature information from photos in each photo group includes: The time information of the photograph is read and normalized to obtain the time element; Based on the GPS coordinates in the EXIF ​​data of the photograph, the location elements are extracted. In the case of missing GPS coordinates, the location elements are extracted by recognizing place names in road signs and banners using OCR and combining this with named entity recognition technology. The facial features of the person are obtained by performing face detection and feature comparison on the photograph. Based on the image content, text, time, location, and people in the photograph, the event elements are derived through event reasoning. Extract the photographer's element from the Artist field of the EXIF ​​data of the photo, the photo watermark OCR, or the device serial number association library.

4. The photo archive organization method according to claim 3, characterized in that... The step of extracting feature information from photos in each photo group also includes: Construct a photo relationship graph for the photos within the photo group; In the event that the time, location, people, event, or photographer elements of the target photo are missing, the target photo is supplemented by supplementing its element information based on the photo relationship graph and by passing messages according to the element information of the neighboring photos of the target photo.

5. A photo archive organization device, characterized in that... ,include: The first processing module is used to group multiple photos based on photo information to obtain at least two photo groups; The second processing module is used to extract feature information and high-level semantic features from the photos in each photo group. The feature information includes time features, location features, people features, event features, and photographer features. The high-level semantic features include global embedding vectors and scene semantic tags. The third processing module is used to generate a photo summary for each photo group based on the element information and the high-level semantic features; The fourth processing module is used to analyze the photo elements of each photo based on the photo summary to obtain the photo title of each photo group; The fifth processing module is used to organize the multiple photos into files based on the photo summary and the photo title; The photo summary includes a structured summary and a natural language summary. The process of generating a photo summary for each photo group based on the element information and the high-level semantic features includes: The element information is fused to obtain the ISL representation of the photograph, and a comprehensive confidence score is calculated for the ISL representation. If the overall confidence level is higher than a preset threshold, the key fields in the ISL representation are filled into a predefined language template to generate the structured summary; If the overall confidence level is lower than or equal to the preset threshold, the ISL representation is converted into a text prompt, and jointly encoded with the global embedding vector and the scene semantic label. The input sequence is fed into the sequence model to generate the natural language summary. Based on the photo summary, the photo elements of each photo are analyzed to obtain the photo title of each photo group, including: Extract summary elements from the photo summary, perform structured parsing on the summary elements, and obtain the photo elements corresponding to the element information; Generate a photo title for the photo based on the photo elements; Common features were obtained by performing commonalities analysis on key fields of each photo in the photo group; Based on the common features, the ISL representations of the photos are weighted and aggregated to generate photo titles for the photo group; The step of generating a photo title based on the photo elements includes: In the visual pathway, the image pixels and cue words of the photograph are input into a multimodal large model to perceive visual content from the image and generate the first title of the photograph; In the ISL pathway, the ISL representation of the photo elements is converted into text prompt words and input into the generation model to generate a second title for the photo; Based on the overall confidence level of the photo, the first title and the second title are merged or filtered to obtain the photo title.

6. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that... When the processor executes the computer program, it implements the photo archive organization method as described in any one of claims 1-4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that... When the computer program is executed by the processor, it implements the photo archive organization method as described in any one of claims 1-4.

Images