AI multimedia retrieval device based on multiple files and combined reading and implementation method

Abstract

The present invention proposes an AI multimedia retrieval device and an implementation method based on multiple files and combined reading, including a content collection module, a content organization module, a content combination module, an AI retrieval module, and a content display module. The files selected by the user are collected and organized to generate a combined file. After AI analysis is performed on the content input by the user, retrieval is performed and the retrieval results are returned. The content display module displays the retrieved results in different modes. The present invention creates a retrieval method that spans various file types such as text, video, picture, voice, dynamic layout files, etc., realizes the retrieval function in a complex file environment, uses AI semantic understanding to remove unnecessary colloquial terms, locates the user's real retrieval intention, and makes the retrieval more accurate; by searching voice, tone, video, audio and other resources from multiple dimensions according to different weight ratios, the retrieval scope is minimized and the accuracy of the retrieval content is improved.

Description

Technical Field

[0001] The present invention relates to the field of AI, and in particular to an AI multimedia retrieval device based on multiple files and combined reading and an implementation method thereof. Background Art

[0002] With the development of society, the demand for retrieval is getting higher and higher in people's daily work and study. In actual use, it is difficult to provide a reasonable and effective retrieval method for file storage when retrieving file content due to the limitations of the directory structure and the number of files. Users cannot customize the selection of one or more files for retrieval at the same time, and the reading effect during retrieval is poor. When retrieving files, usually only text content retrieval is supported, the retrieval mode is single and incomplete, and it does not support AI retrieval corresponding to the file and corresponding jump to the retrieval content, resulting in a poor retrieval experience. Summary of the invention

[0003] The purpose of the present invention is to provide an AI multimedia retrieval device and implementation method based on multiple files and combined reading to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solution: an AI multimedia retrieval device based on multi-file and combined reading, characterized in that it includes a content collection module, a content arrangement module, a content combination module, an AI retrieval module, and a content display module.

[0005] The content collection module collects files of multiple different formats selected by the user and classifies the files according to different file formats;

[0006] The content sorting module sorts the files according to the classification of the content collection module, performs format verification on the files, performs corresponding data processing on files of different formats, and performs vector analysis on the files;

[0007] The content combination module combines part or all of the files selected by the user as needed, and forms the file contents as a whole into a combined file or into a mixture of part of the combined file and independent files;

[0008] The AI ​​retrieval module performs AI analysis based on the content input by the user, retrieves the input content from the file queue in a multi-threaded manner and returns the retrieval results;

[0009] The content display module displays the retrieved results in different modes, marks each piece of the retrieved content, creates upper and lower index jumps for different pages and formats, jumps independent files within the file, and jumps multiple files between files.

[0010] Preferably, the file formats collected by the content collection module include but are not limited to wps, doc / docx, xls / xlsx, ppt / pptx, cad, jpg, tif, gif, png, pdf, html, txt, true, DLF, ofd, video, and audio.

[0011] Preferably, the content arrangement module processes various input files. When the content arrangement module processes video and audio files, it first performs text recognition on the screen content in the video, and separates the voice separately, recognizes the corresponding text in the voice, and keeps the text content consistent with the video according to the time vector, maps and matches the video screen with the content divided by time, and forms a file in the format of a layout data stream;

[0012] When processing the scanned document or image, the content arrangement module performs OCR operation on it to generate a double-layer PDF file.

[0013] Preferably, when combining multiple continuous video files, the content combination module segments different video files in the X direction, and the time scale of playback is a double-bar directory;

[0014] The content combination module combines continuous image files using 1*1, 2*2, and 3*3 formats for typesetting. When the mouse is slid and floated for reading, not only the name of the image can be displayed, but also the image can be enlarged for reading.

[0015] The combined reading module supports code reading when combining files of different formats for display.

[0016] Preferably, when the content input by the AI ​​search module is text, keyword matching is performed with the text file, searching is performed in the video and picture files, and content matching the searched text is selected from the video and picture library;

[0017] When the content input by the AI ​​retrieval module is a picture, the picture detector picture data to be retrieved is used to extract picture features, the picture data is detected in the image or video to be searched, and the picture vector is extracted, and the similarity between the picture vector to be detected and the extracted picture vector is calculated, and whether the same picture content is determined based on the result of the similarity calculation and the set threshold value;

[0018] When the content input by the AI ​​retrieval module is speech, the text content and the timbre of the speech are first matched with the combined file, and then the content matching retrieval is performed according to the converted text;

[0019] When the content input by the AI ​​retrieval module is a video, the text, voice, and picture in the input video are separated separately, different retrieval weights are set, and the combined file is searched.

[0020] Preferably, when the AI ​​retrieval module performs search and matching, it performs text matching search for pure text files; sets weights for text and pictures for content with text and pictures, and performs matching search based on the weights; performs text and tone matching search for voice; and uses a variety of methods to perform weight analysis for videos, and performs matching search based on the set weights; when searching, the AI ​​retrieval module supports users to choose to filter files whose content is not matched.

[0021] Preferably, the content display module splices the content queue returned by the search, and creates a one-to-one jump matching relationship between the content queue and the original text, creates an index relationship between the previous and next content, and makes different marks for different query contents;

[0022] When the matching content displayed by the content display module is a picture, the picture supports highlighting, and an association relationship is created on the locked content, and the layout is displayed in a 1*1, 2*2, or 3*3 manner, and a transparent layer is added to the original picture to display the matched content;

[0023] When the matching content displayed by the content display module is a video, the matching content is displayed accurately to the frame.

[0024] The AI ​​multimedia retrieval implementation method based on multiple files and combined reading is characterized by comprising the following steps:

[0025] Step S1: Upload files. After the user selects the target file, the content collection module collects the files and classifies them according to different file formats;

[0026] Step S2: the content arrangement module performs corresponding data processing on files of different formats, and performs vector analysis, text extraction, and conversion operations on the files;

[0027] Step S3: the content combination module combines part or all of the files as required according to the user's selection, so that the file content as a whole forms a combined file or a mixture of a partial combined file and an independent file;

[0028] Step S4: The user selects the file range to be searched and sets the corresponding display mode, and displays the files in the set form;

[0029] Step S5: construct an input box, and input text, voice, video or mixed content information in the input box;

[0030] Step S6: Perform weighted matching search within the selected file range, create an index for the retrieved content and associate it with the previous and next content, and build a quick reading guide page;

[0031] Step S7: Combine the retrieved contents and display them in summary as required.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) The present invention creates a retrieval method for independent files or combined files or a mixture of independent files and combined files, and the file types span various file types such as text, video, picture, voice, dynamic layout file (DLF), etc.;

[0034] (2) The present invention provides retrieval methods such as text, video, picture, voice, and timbre in the above environment, realizing the retrieval function in a complex file environment;

[0035] (3) The present invention uses AI semantic understanding to remove unnecessary colloquial terms, locate the user's real search intention, and make the search more accurate; by searching voice, tone, video, audio and other resources from multiple dimensions according to different weight ratios, the search scope is minimized and the accuracy of the search content is improved. DETAILED DESCRIPTION

[0036] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The AI ​​multimedia retrieval device based on multiple files and combined reading includes a content collection module, a content arrangement module, a content combination module, an AI retrieval module, and a content display module;

[0038] The content collection module collects files of various formats selected by the user and classifies the files according to different file formats, including but not limited to wps, doc / docx, xls / xlsx, ppt / pptx, cad, jpg, tif, gif, png, pdf, html, txt, true, DLF, ofd, video, audio, where the DLF file is a compressed file, which contains a navigation file and several OFD files, where the navigation file mainly records the association relationship between files, the entry of files, the storage path of OFD files, the triggering position and jump events between OFD and OFD;

[0039] The content sorting module sorts the files according to the classification of the content collection module, performs format verification on the files, performs corresponding data processing on files of different formats, and performs vector analysis on the files;

[0040] The content combination module combines part or all of the files selected by the user as needed, and forms the file contents as a whole into a combined file or into a mixture of part of the combined file and independent files;

[0041] The AI ​​retrieval module performs AI analysis based on the content input by the user, retrieves the input content from the file queue in a multi-threaded manner and returns the retrieval results;

[0042] The content display module displays the search results in different modes, marks the searched content in each place, creates upper and lower index jumps for different pages and formats, jumps independent files within the file, and jumps multiple files between files. Example

[0043] The AI ​​multimedia retrieval implementation method based on multiple files and combined reading has the following implementation steps:

[0044] Step S1: Upload files. After the user selects a target file, the target file format selected by the user supports one or more formats. The content collection module collects the files and classifies them according to different file formats.

[0045] Step S2: The content arrangement module performs corresponding data processing on files of different formats, and performs vector analysis, text extraction, and conversion operations on the files. When processing video and audio files, firstly, the text content in the video is recognized, and the voice is separated separately to identify the corresponding text in the voice. Then, according to the time vector, the text content is kept consistent with the video, and the video screen is mapped and matched with the content divided by time to form a file in the format of a layout data stream;

[0046] When the content arrangement module processes the scanned documents or images, it performs OCR operations on them and generates a double-layer PDF file;

[0047] Step S3: the content combination module combines part or all of the files as required according to the user's selection, so that the file content as a whole forms a combined file or a mixture of a partial combined file and an independent file;

[0048] When combining multiple continuous video files, the different video files are segmented in the X direction, and the playback time scale is made into a double-bar directory;

[0049] For the combination of continuous image files, different formats are used for typesetting in the form of 1*1, 2*2, and 3*3. When the mouse is moved to float reading, when the mouse is moved to a specific image, not only the name of the current image can be displayed, but also the current image can be enlarged for reading;

[0050] When files of different formats are combined and displayed, code reading is supported.

[0051] Step S4: The user selects the file range to be searched and sets the corresponding display mode, and displays the files in the set form;

[0052] Step S5: construct an input box, and input text, voice, video or mixed content information in the input box;

[0053] When the input content is text, the AI ​​search module matches keywords with the text file, searches in the video and image files, and selects content that matches the search text from the video and image library;

[0054] When the input content is a picture, the AI ​​retrieval module extracts picture features based on the picture data of the picture detector to be retrieved, detects the picture data in the image or video to be searched, and extracts the picture vector, calculates the similarity between the picture vector to be detected and the extracted picture vector, and determines whether it is the same picture content based on the result of the similarity calculation and the set threshold;

[0055] When the input content is voice, the AI ​​retrieval module first matches the text content and voice timbre of the voice with the combined file, and then performs content matching retrieval based on the converted text;

[0056] When the input content is a video, the AI ​​retrieval module will separate the text, voice, and picture in the input video, set different retrieval weights, and search the combined file;

[0057] Step S6: The AI ​​retrieval module performs a weighted matching search within the selected file range, creates an index for the retrieved content and associates them with each other, and constructs a quick reading guide page;

[0058] Among them, when the AI ​​retrieval module performs search and matching, it performs text matching search for pure text files;

[0059] For content with text and pictures, set weights for the text and pictures, and search based on the weights;

[0060] For voice, we perform matching retrieval based on text and tone; for video, we use multiple methods to perform weight analysis and perform matching retrieval based on the set weights;

[0061] AI semantic understanding is used to remove unnecessary colloquial terms, locate the user's true search intent, and make the search more accurate; by searching voice, tone, video, audio and other resources from multiple dimensions at different weight ratios, the search scope is minimized and the accuracy of the search content is improved.

[0062] Step S7: The content display module splices the content queue returned by the search, and creates a one-to-one jump matching relationship between it and the original text, creates an index relationship between the previous and next contents, and makes different marks for different query contents;

[0063] When the matching content displayed by the content display module is a picture, the picture supports highlighting, and creates an association relationship with the locked content, using 1*1, 2*2, 3*3 for layout display, and adding a transparent layer on the original picture to display the matched content;

[0064] When the matching content displayed by the content display module is a video, the matching content is displayed accurately to the frame.

[0065] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and scope of the equivalent elements of the claims be included in the invention. Any marking in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An AI multimedia retrieval device based on multiple files and combined reading, characterized in that: It includes content collection module, content organization module, content combination module, AI retrieval module and content display module. The content collection module collects files of multiple different formats selected by the user and classifies the files according to different file formats; The content arrangement module arranges the files according to the classification of the content collection module, performs format verification on the files, performs corresponding data processing on files of different formats, and performs vector analysis on the files; when the content arrangement module processes the video and audio files, it first performs text recognition on the screen content in the video, and separates the voice separately, recognizes the corresponding text in the voice, and keeps the text content consistent with the video according to the time vector, maps and matches the video screen with the content divided by time, and forms a file in the format of a layout data stream; When the content arrangement module processes the scanned document or image, it performs OCR operation on it to generate a double-layer PDF file; The content combination module combines part or all of the files selected by the user as needed, and forms a combined file as a whole or a mixture of a partial combined file and an independent file. When the content combination module combines multiple continuous video files, it segments different video files in the X direction, and the playback time scale is a double-bar directory. When the content combination module combines continuous picture files, it uses 1*1, 2*2, 3*3 to layout in different formats. When the mouse is slid and floated for reading, not only the name of the picture can be displayed, but also the picture can be enlarged for reading. When the content combination module combines files of different formats for display, it supports riding code reading. The AI ​​retrieval module performs AI analysis based on the content input by the user, retrieves the input content from the file queue in a multi-threaded manner and returns the retrieval results; wherein, when the content input by the AI ​​retrieval module is text, it matches keywords with the text file, searches in the video and picture files, and selects content matching the searched text from the video and picture library; When the content input by the AI ​​retrieval module is a picture, the picture data is detected according to the picture to be retrieved, the picture features are extracted, the picture data is detected in the image or video to be searched, and the picture vector is extracted, and the similarity between the picture vector to be detected and the extracted picture vector is calculated, and whether it is the same picture content is determined according to the result of the similarity calculation and the set threshold value; when the content input by the AI ​​retrieval module is voice, the text content and the timbre of the voice are first matched with the combined file, and then the content matching retrieval is performed according to the converted text; when the content input by the AI ​​retrieval module is a video, the text, voice, and picture in the input video are stripped separately, different retrieval weights are set, and the combined file is retrieved; when the AI ​​retrieval module performs retrieval and matching, for pure text files, text matching retrieval is performed; for content with text and pictures, the weights of text and pictures are set, and the matching retrieval is performed according to the weights; for voice, a matching retrieval of text and tone is performed; for video, a variety of methods are used for weight analysis, and matching retrieval is performed according to the set weights; when the AI ​​retrieval module searches, it supports users to choose to filter files that do not match the content; The content display module displays the retrieved results in different modes, marks each piece of the retrieved content, creates upper and lower index jumps for different pages and formats, jumps independent files within the file, and jumps multiple files between files.

2. The AI ​​multimedia retrieval device based on multiple files and combined reading according to claim 1 is characterized in that: The file formats collected by the content collection module include but are not limited to wps, doc / docx, xls / xlsx, ppt / pptx, cad, jpg, tif, gif, png, pdf, html, txt, true, DLF, ofd, video, and audio.

3. The AI ​​multimedia retrieval device based on multiple files and combined reading according to claim 1 is characterized in that: The content display module splices the content queue returned by the search, creates a one-to-one jump matching relationship between the content and the original text, creates an index relationship between the previous and next content, and makes different marks for different query contents; When the matching content displayed by the content display module is a picture, the picture supports highlighting, and creates an association relationship with the locked content, and uses 1*1, 2*2, 3*3 to display the layout, and adds a transparent layer on the original picture to display the matched content; When the matching content displayed by the content display module is a video, the matching content is displayed accurately to the frame.

4. The retrieval method of the AI ​​multimedia retrieval device based on multiple files and combined reading according to claim 1 is characterized in that The following steps are involved: Step S1: Upload files. After the user selects the target file, the content collection module collects the files and classifies them according to different file formats; Step S2: the content arrangement module performs corresponding data processing on files of different formats, and performs vector analysis, text extraction, and conversion operations on the files; Step S3: the content combination module combines part or all of the files as required according to the user's selection, so that the file contents as a whole form a combined file or a mixture of a partial combined file and an independent file; Step S4: The user selects the file range to be searched and sets the corresponding display mode, and displays the files in the set form; Step S5: construct an input box, and input text, voice, video or mixed content information in the input box; Step S6: Perform weighted matching search within the selected file range, create an index for the retrieved content and associate it with the previous and next content, and build a quick reading guide page; Step S7: Combine the retrieved contents and display them in summary as required.