File processing method and device, equipment and storage medium

Abstract

The invention provides a file processing method, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, natural language processing, deep learning, large models and the like. The method comprises the following steps: extracting browsing information corresponding to browsing attributes from a to-be-processed file; extracting content information from the to-be-processed file; splicing the browsing information and the content information to obtain a target prompt word; and inputting the to-be-processed file and the target cue word into the large model, and outputting to obtain a target fragment of the to-be-processed file. According to the method, the generation efficiency of the wonderful fragment (namely the target fragment) is improved, the pertinence and accuracy of the wonderful fragment are improved, and the user experience is further improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as computer vision, natural language processing, deep learning, and large models, and in particular to a file processing method, device, equipment, and storage medium. Background Art

[0002] In current multimedia content consumption, users often want to jump directly to the most exciting and thrilling part. For example, when a user is listening to music, he may want to jump directly to the chorus. For another example, when a user is watching a video, he may want to jump directly to the most exciting part of the video. Summary of the invention

[0003] The present disclosure provides a file processing method, apparatus, device and storage medium.

[0004] According to a first aspect of the present disclosure, a file processing method is provided, including: extracting browsing information corresponding to browsing attributes from a file to be processed; extracting content information from the file to be processed; splicing the browsing information and the content information to obtain a target prompt word; inputting the file to be processed and the target prompt word into a large model, and outputting a target segment of the file to be processed.

[0005] According to a second aspect of the present disclosure, there is provided a file processing device, comprising: a browsing information extraction module, configured to extract browsing information corresponding to browsing attributes from a file to be processed; a content information extraction module, configured to extract content information from a file to be processed; a splicing module, configured to splice browsing information and content information to obtain a target prompt word; and an output module, configured to input the file to be processed and the target prompt word into a large model, and output a target segment of the file to be processed.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner in the first aspect.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method described in any implementation manner in the first aspect.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0012] Figure 2 is a flowchart of an embodiment of a file processing method according to the present disclosure;

[0013] Figure 3 is a flowchart of another embodiment of the file processing method according to the present disclosure;

[0014] Figure 4 is a flowchart of another embodiment of the file processing method according to the present disclosure;

[0015] Figure 5 is a flowchart of another embodiment of the file processing method according to the present disclosure;

[0016] Figure 6 is an application scenario diagram of the file processing method according to the present disclosure;

[0017] Figure 7 is a structural schematic diagram of an embodiment of a file processing device according to the present disclosure;

[0018] Figure 8 It is a block diagram of an electronic device used to implement the file processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] Figure 1An exemplary system architecture 100 is shown to which an embodiment of the file processing method or file processing apparatus of the present disclosure can be applied.

[0022] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send information, etc. Various client applications can be installed on the terminal devices 101, 102, 103.

[0024] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the above electronic devices. They may be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0025] The server 105 can provide various services. For example, the server 105 can analyze and process the to-be-processed files obtained from the terminal devices 101, 102, 103, and generate processing results (eg, target segments).

[0026] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0027] It should be noted that the file processing method provided in the embodiment of the present disclosure is generally executed by the server 105 , and accordingly, the file processing device is generally disposed in the server 105 .

[0028] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0029] Continue to refer Figure 2, which shows a process 200 of an embodiment of a file processing method according to the present disclosure. The file processing method comprises the following steps:

[0030] Step 201: extract browsing information corresponding to the browsing attribute from the file to be processed.

[0031] In this embodiment, the execution subject of the file processing method (for example Figure 1 The server 105 shown in the figure will extract browsing information corresponding to the browsing attribute from the to-be-processed file. Here, the to-be-processed file may refer to a music file or a video file, and the video file may be a video part extracted from a movie or a TV series, or may be a live real-time video or a video clip of a live stream playback. This embodiment does not limit the specific type of the to-be-processed file.

[0032] Browsing attributes refer to some attribute information of the file to be processed in real time, such as the number of real-time browsers, real-time comments, barrages, likes information, etc. When the file to be processed is a live video, the browsing information can also include the number of people entering the live room in real time, the number of people leaving the live room in real time, etc.

[0033] Optionally, after extracting the real-time browsing information, the above-mentioned execution entity will also convert it into structured data. Structured data is also called row data, which is data logically expressed and implemented by a two-dimensional table structure, strictly follows the data format and length specifications, and is mainly stored and managed through a relational database.

[0034] Step 202: extract content information from the file to be processed.

[0035] In this embodiment, the execution subject extracts content information from the file to be processed. The content information is the data content information in the file, which is a kind of unstructured data. The content information may include audio information, text information, picture information, etc. The audio information may specifically include spectrum, rhythm, etc. The text information may specifically include keywords, emotional words, etc. The picture information may specifically include picture brightness information, picture scene information, etc. When the content information includes more than two of the audio information, text information, and picture information, the execution subject will fuse the extracted content information after extracting the content information, thereby obtaining the fused multimodal feature information, thereby obtaining the fused content information.

[0036] Step 203: combine the browsing information and the content information to obtain the target prompt word.

[0037] In this embodiment, the execution subject will splice the extracted browsing information and content information, that is, splice and combine the structured data and unstructured data extracted above, so as to obtain the target prompt word. Prompt is the input sentence or phrase provided by the user to the model when using the artificial intelligence model, the purpose is to guide the model to generate a specific type or style of output. In simple terms, the prompt word is an instruction to the model, telling the model what kind of answer it wants to get or what kind of text, picture or video it wants to generate.

[0038] Step 204: input the file to be processed and the target prompt word into the large model, and output the target segment of the file to be processed.

[0039] In this embodiment, the above-mentioned execution subject will splice the target prompt words and the file to be processed into the big model, so that the big model can make an analysis and decision, and output the target segment of the file to be processed, which is the wonderful segment in the file to be processed or the segment that the user is most interested in. Since the target prompt words have integrated comment information, barrage information, audio information, video screen information, text information, etc., the big model can analyze the file to be processed based on the target prompt words, so as to extract the most wonderful segment from the file to be processed, that is, the target segment. The target segment can be one or more, that is, the big model can output top1 or top K (K is an integer) wonderful segments.

[0040] It should be noted that a large language model (large model) is an artificial intelligence model designed to understand and generate human language. By training on large amounts of text data, it can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. A large model refers to a deep neural network with more than 1 billion parameters that can process massive amounts of data and complete various complex tasks. With the continuous improvement of computer hardware performance and the continuous optimization of deep learning algorithms, the development of large models is also accelerating. The parameter scale of large models continues to expand, the training time is getting longer and longer, but the performance is also improving. Large models are usually based on deep learning architectures, such as transformers, which enable impressive capabilities in various natural language processing tasks. Common large models can include but are not limited to ChatGPT, GPT-4, ERNIE Bot (Wenxin Yiyan), etc.

[0041] The file processing method provided by the disclosed embodiment first extracts browsing information corresponding to the browsing attribute from the file to be processed; then extracts content information from the file to be processed; then splices the browsing information and content information to obtain the target prompt word; finally, the file to be processed and the target prompt word are input into the large model, and the target segment of the file to be processed is obtained by output. The file processing method in this embodiment obtains the target prompt word by integrating the browsing information (such as comment information, barrage information, etc.) and content information (such as audio information, video screen information, text information, etc.) in the file to be processed, so that the large model outputs the highlight segment (i.e., the target segment) of the file to be processed based on the target prompt word, thereby improving the efficiency of generating the highlight segment, and improving the pertinence and accuracy of the highlight segment, further improving the user experience.

[0042] In some optional implementations of this embodiment, the file processing method further includes: detecting the content integrity of the target segment, and adjusting the interval value of the time interval of the target segment based on the detection result.

[0043] In this implementation, after the target segment is generated, the content integrity of the target segment is also detected, that is, whether the target segment is a complete and continuous content. If not, the interval value of the time interval of the target video needs to be adjusted, such as adjusting the start time and / or end time of the target segment. The start and end time of the target segment can also be fine-tuned according to the context information to ensure the integrity and coherence of the segment. In addition, repeated segments in the target segment can also be identified and filtered, so as to ensure the diversity of multimedia content on the basis of ensuring the content integrity of the target segment, thereby improving the user experience.

[0044] In some optional implementations of this embodiment, the file processing method further includes: obtaining user feedback information on the target segment; and adjusting model parameters of the large model based on the feedback information.

[0045] In this implementation, the execution subject will also obtain user feedback on the target segment, which indicates whether the user thinks the target segment is a wonderful segment in the file to be processed. Then, the user feedback information is analyzed using data analysis technology, and the parameters of the large model are adjusted and optimized based on the analysis results, so that the large model can output more accurate segment content, that is, improve the accuracy of the large model.

[0046] In addition, in the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0047] Continue to refer Figure 3 , Figure 3A process 300 of another embodiment of a file processing method according to the present disclosure is shown. The file processing method comprises the following steps:

[0048] Step 301: extract browsing data corresponding to the browsing attribute from the file to be processed.

[0049] In this embodiment, the execution subject of the file processing method (for example Figure 1 The server 105 shown in the figure will extract browsing data corresponding to the browsing attribute from the to-be-processed file, wherein the browsing data includes: real-time traffic data and / or real-time comment data. The real-time traffic data may include data information such as the real-time number of browsers, the real-time number of new users, and the real-time number of people leaving, and the real-time comment data may include data information such as real-time bullet screen information, real-time comment information, and real-time like information.

[0050] Step 302: convert the browsing data into structured data in a preset format.

[0051] In this embodiment, the above-mentioned execution subject converts the browsing data into structured data in a preset format. Here, the extracted real-time browsing data can be converted into a data line chart for more intuitive browsing, and then the browsing data can be converted into structured data in a preset format. The preset format can be JSON (JavaScript Object Notation) format or other formats. Structured data is also called row data. It is data that is logically expressed and implemented by a two-dimensional table structure, strictly follows the data format and length specifications, and is mainly stored and managed through a relational database. By converting the browsing data into structured data in a preset format, the processing efficiency of subsequent large models can be improved.

[0052] Step 303, performing shot segmentation on the picture data of the file to be processed to obtain a plurality of segmented shot pictures.

[0053] In this embodiment, the content information includes picture information, and the execution subject will perform shot segmentation on the picture data, that is, segment multiple continuous pictures according to different scenes, so as to obtain multiple segmented shot pictures, each of which may include one or more frames, and these multiple frames correspond to the same scene, so as to analyze each shot separately later.

[0054] Step 304: perform image analysis on the lens image to determine the key frames in the lens image.

[0055] In this embodiment, the execution subject will perform image analysis on the lens picture to determine the key frame in the lens picture. The key frame is the picture frame representing the main content, and the key frame can be extracted by using analysis technology.

[0056] Step 305: extracting picture information from the key frame.

[0057] In this embodiment, the execution subject extracts picture information from the key frame, wherein the picture information includes at least one of the following: picture brightness information, picture scene information, and picture area information. That is, the execution subject extracts features such as brightness change, scene switching, and visual attention area of ​​the key frame, and uses the extracted information as the picture information of the shot.

[0058] By determining the key frames, the image information of the current shot can be obtained more accurately and quickly.

[0059] Step 306: extract audio information from the audio data of the file to be processed.

[0060] In this embodiment, the content information also includes audio information, and the execution subject will also extract audio information from the audio data of the file to be processed, wherein the audio information includes at least one of the following: volume information, spectrum information, and rhythm information. That is, the execution will also extract audio features, including extracting features such as volume, spectrum, loudness, and rhythm. Thus, the audio information can be accurately and quickly acquired.

[0061] Step 307: perform natural language processing on the text content of the file to be processed to determine text information corresponding to the text content.

[0062] In this embodiment, the content information also includes text information. The execution subject will also perform natural language processing on the text content of the file to be processed to determine the text information corresponding to the text content, wherein the text information includes at least one of the following: keywords, sentiment words, and target paragraph information. The execution subject will perform natural language processing on the subtitles or lyrics to extract information such as sentiment words, keywords, repeated words or repeated paragraphs, so as to accurately and quickly obtain the text information.

[0063] Step 308: fuse the image information, audio information, and text information to obtain a fused first multimodal feature.

[0064] In this embodiment, the above-mentioned execution entity will fuse the extracted picture information, audio information and text information to obtain a fused first multimodal feature.

[0065] Step 309: Concatenate the structured data and the first multimodal feature to obtain a target prompt word.

[0066] In this embodiment, the execution subject will splice the structured data and the first multimodal feature to obtain the target prompt word. Thus, by first fusing the image, audio, and text information of the file to be processed, and then splicing the structured data and the unstructured data (the first multimodal feature) to obtain the target prompt word, the decision accuracy of the subsequent large model can be improved.

[0067] Step 310: input the file to be processed and the target prompt word into the large model, and output the target segment of the file to be processed.

[0068] Step 310 is basically consistent with step 204 of the aforementioned embodiment. The specific implementation method can refer to the aforementioned description of step 204, which will not be repeated here.

[0069] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiment, the file processing method in this embodiment first integrates the picture, audio, and text information of the file to be processed to obtain the first multimodal feature, and then splices the above-mentioned structured data and unstructured data (first multimodal feature) to obtain the target prompt word, thereby improving the decision-making accuracy of the large model.

[0070] Continue to refer Figure 4 , Figure 4 A process 400 of another embodiment of a file processing method according to the present disclosure is shown. The file processing method comprises the following steps:

[0071] Step 401: extract browsing data corresponding to the browsing attribute from the file to be processed.

[0072] Step 402: convert the browsing data into structured data in a preset format.

[0073] Step 403, performing shot segmentation on the picture data of the file to be processed to obtain a plurality of segmented shot pictures.

[0074] Step 404: perform image analysis on the lens image to determine the key frames in the lens image.

[0075] Step 405: extracting picture information from the key frame.

[0076] Step 406: extract audio information from the audio data of the file to be processed.

[0077] Step 407: perform natural language processing on the text content of the file to be processed to determine text information corresponding to the text content.

[0078] Steps 401-407 are basically consistent with steps 301-307 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 301-307, which will not be repeated here.

[0079] Step 408: determine a key segment from the music file based on the frequency spectrum information, and obtain target audio information of the key segment.

[0080] In this embodiment, the files to be processed include music files, and the execution subject of the file processing method (for example Figure 1 The server 105 shown in the figure will extract the music spectrum rhythm content of the music file, determine the key segment based on the spectrum information, and then obtain the audio information of the key segment, that is, the target audio information, so as to accurately and quickly determine the key segment in the music.

[0081] In some optional implementations of this embodiment, step 408 includes: determining multiple peak points and trough points based on spectrum information, and calculating the difference between the peak points and the trough points; determining target peak points and target trough points whose differences are greater than a preset threshold; and determining the music clips corresponding to the target peak points and target trough points as key clips.

[0082] In this implementation, the execution subject determines multiple peaks and troughs based on the spectrum information, and calculates the difference between each peak and each trough, thereby extracting the portion with a large difference between peaks and troughs, that is, the segment between the target peak and target trough with a difference greater than a preset threshold is taken as the key segment. Thus, the key segment is determined from the music file by the peak and trough method.

[0083] In some optional implementations of this embodiment, the file processing method further includes: determining key segments from the music file based on the user's behavioral characteristics, where the user's behavioral characteristics may be the user's behavior of staying or leaving, and obtaining the user's stay time point and / or the user's leave data time point, and determining the key segments based on the behavioral characteristics. For example, if the user stays, it can be considered that the user is interested, so it can be preliminarily considered that the content is relatively key; if the user leaves, it can be considered that the user is not interested, so it can be preliminarily classified as non-key content. Thus, the key segments of the music file can be quickly and accurately determined through the user's behavioral characteristics.

[0084] Step 409: fuse the picture information, the target audio information and the text information to obtain a fused second multimodal feature.

[0085] In this embodiment, the execution entity will fuse the picture information, the target audio information and the text information to obtain a fused second multimodal feature.

[0086] Step 410: Concatenate the structured data and the second multimodal features to obtain a target prompt word.

[0087] In this embodiment, the execution subject will splice the structured data and the second multimodal feature to obtain the target prompt word. Thus, by first fusing the image, audio, and text information of the file to be processed, and then splicing the structured data and the unstructured data (the second multimodal feature) to obtain the target prompt word, the decision accuracy of the subsequent large model can be improved.

[0088] Step 411, input the file to be processed and the target prompt word into the large model, and output the target segment of the file to be processed.

[0089] Step 411 is basically consistent with step 204 of the aforementioned embodiment. The specific implementation method can refer to the aforementioned description of step 204, which will not be repeated here.

[0090] from Figure 4 It can be seen that Figure 3 Compared with the corresponding embodiment, the file processing method in this embodiment integrates the key segment audio information in the music file as the target audio information to obtain the second multimodal feature, and then splices the above-mentioned structured data and unstructured data (second multimodal feature) to obtain the target prompt word, thereby improving the decision accuracy of the large model.

[0091] Continue to refer Figure 5 , Figure 5 A process 500 of another embodiment of a file processing method according to the present disclosure is shown. The file processing method comprises the following steps:

[0092] Step 501: extract browsing data corresponding to the browsing attribute from the file to be processed.

[0093] Step 502: convert the browsing data into structured data in a preset format.

[0094] Step 503, performing shot segmentation on the picture data of the file to be processed to obtain a plurality of segmented shot pictures.

[0095] Step 504: perform image analysis on the lens image to determine the key frames in the lens image.

[0096] Step 505: extracting picture information from the key frame.

[0097] Step 506: extract audio information from the audio data of the file to be processed.

[0098] Step 507: perform natural language processing on the text content of the file to be processed to determine text information corresponding to the text content.

[0099] Steps 501-507 are basically consistent with steps 301-307 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 301-307, which will not be repeated here.

[0100] Step 508: divide the text content according to preset time levels to obtain multiple sub-texts, and score the sub-texts to generate score information of the sub-texts.

[0101] In this embodiment, the file to be processed includes a video file, and the execution subject of the file processing method (for example Figure 1 The server 105 shown in the figure will divide the text content according to the preset time level, for example, divide it according to the minute level, so as to obtain multiple sub-texts. Then the sub-texts are scored respectively, so as to generate the score information of the sub-texts. Specifically, the sub-texts can be scored using a pre-trained scoring model, and the score information obtained can be a specific score (for example, 80 points) or a binary classification information (for example, wonderful or not wonderful). Thus, the content of each sub-text is evaluated.

[0102] Step 509: fuse the picture information, audio information, text information and score information to obtain a fused third multimodal feature.

[0103] In this embodiment, the execution entity will fuse the picture information, audio information, text information and score information to obtain a fused third multimodal feature.

[0104] Step 510: Concatenate the structured data and the third multimodal feature to obtain a target prompt word.

[0105] In this embodiment, the execution subject will splice the structured data and the third multimodal feature to obtain the target prompt word. Thus, by scoring the content of each sub-text, the image, audio, text, and score information of the file to be processed are integrated to obtain the third multimodal feature, and then the structured data and unstructured data (third multimodal feature) are spliced ​​to obtain the target prompt word, thereby improving the decision accuracy of the large model.

[0106] Step 511, input the file to be processed and the target prompt word into the large model, and output the target segment of the file to be processed.

[0107] Step 511 is basically consistent with step 204 of the aforementioned embodiment. The specific implementation method can refer to the aforementioned description of step 204, which will not be repeated here.

[0108] from Figure 5 It can be seen that Figure 4Compared with the corresponding embodiment, the file processing method in this embodiment also scores the content of each sub-text, integrates the picture, audio, text, and score information of the file to be processed to obtain a third multimodal feature, and then splices the above-mentioned structured data and unstructured data (third multimodal feature) to obtain the target prompt word, thereby improving the decision-making accuracy of the large model.

[0109] Continue to refer Figure 6 , Figure 6 An application scenario of the file processing method according to the present disclosure is shown, and the application scenario is a live broadcast scenario. In this application scenario, the above-mentioned execution subject will first extract the real-time number of online people, the real-time number of people leaving, the real-time number of transactions, the real-time comment data, the real-time bullet screen data, and the order conversion data of the file to be processed, and convert the extracted data into structured data. Then extract the video and audio spectrum rhythm and video spectrum of the file to be processed, score the minute-level video live text content, extract emotional words, keywords, repeated paragraphs and other information, and generate unstructured data based on the extracted data. After that, splice the above-mentioned structured data and unstructured data to obtain the target prompt word prompt. Finally, the spliced ​​prompt is input into the large model so that the large model can make analysis decisions and output the top1 or top K highlights.

[0110] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a file processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0111] like Figure 7 As shown, the file processing device 700 of this embodiment includes: a browsing information extraction module 701, a content information extraction module 702, a splicing module 703 and an output module 704. Among them, the browsing information extraction module 701 is configured to extract browsing information corresponding to the browsing attribute from the file to be processed; the content information extraction module 702 is configured to extract content information from the file to be processed; the splicing module 703 is configured to splice the browsing information and the content information to obtain the target prompt word; the output module 704 is configured to input the file to be processed and the target prompt word into the large model, and output to obtain the target segment of the file to be processed.

[0112] In this embodiment, the specific processing of the browsing information extraction module 701, the content information extraction module 702, the splicing module 703 and the output module 704 and the technical effects thereof can be referred to in the respective Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0113] In some optional implementations of this embodiment, the browsing information extraction module is further configured to: extract browsing data corresponding to the browsing attributes from the file to be processed, wherein the browsing data includes: real-time traffic data and / or real-time comment data; and convert the browsing data into structured data in a preset format.

[0114] In some optional implementations of this embodiment, the content information includes picture information; and the content information extraction module is further configured to: perform lens segmentation on the picture data of the file to be processed to obtain multiple segmented lens pictures; perform image analysis on the lens pictures to determine the key frames in the lens pictures; extract picture information from the key frames, wherein the picture information includes at least one of the following: picture brightness information, picture scene information, and picture area information.

[0115] In some optional implementations of this embodiment, the content information also includes audio information; and the content information extraction module is further configured to: extract audio information from audio data of the file to be processed, wherein the audio information includes at least one of the following: volume information, spectrum information, and rhythm information.

[0116] In some optional implementations of this embodiment, the content information also includes text information; and the content information extraction module is further configured to: perform natural language processing on the text content of the file to be processed, and determine the text information corresponding to the text content, wherein the text information includes at least one of the following: keywords, sentiment words, and target paragraph information.

[0117] In some optional implementations of this embodiment, the above-mentioned file processing device 700 also includes: a first fusion module, configured to fuse the picture information, audio information and text information to obtain a fused first multimodal feature; and the splicing module is further configured to: splice the structured data and the first multimodal feature to obtain the target prompt word.

[0118] In some optional implementations of this embodiment, the file to be processed includes a music file; and the content information extraction module includes: a segment determination submodule, configured to determine a key segment from the music file based on spectrum information; and an information acquisition submodule, configured to acquire target audio information of the key segment.

[0119] In some optional implementations of this embodiment, the segment determination submodule is further configured to: determine multiple peak points and trough points based on spectrum information, and calculate the difference between the peak points and the trough points; determine target peak points and target trough points whose differences are greater than a preset threshold; and determine the music segments corresponding to the target peak points and target trough points as key segments.

[0120] In some optional implementations of this embodiment, the file processing device 700 further includes: a segment determination module configured to determine key segments from the music file based on the user's behavioral characteristics.

[0121] In some optional implementations of this embodiment, the above-mentioned file processing device 700 also includes: a second fusion module, configured to fuse the picture information, target audio information and text information to obtain a fused second multimodal feature; and the splicing module is further configured to: splice the structured data and the second multimodal feature to obtain the target prompt word.

[0122] In some optional implementations of this embodiment, the file to be processed includes a video file; and the content information extraction module is further configured to: divide the text content according to preset time levels to obtain multiple sub-texts; score the sub-texts and generate sub-text score information.

[0123] In some optional implementations of this embodiment, the above-mentioned file processing device 700 also includes: a third fusion module, configured to fuse the picture information, audio information, text information and score information to obtain a fused third multimodal feature; and the splicing module is further configured to: splice the structured data and the third multimodal feature to obtain the target prompt word.

[0124] In some optional implementations of this embodiment, the file processing device 700 further includes: a detection module configured to detect the content integrity of the target segment and adjust the interval value of the time interval of the target segment based on the detection result.

[0125] In some optional implementations of this embodiment, the file processing device 700 further includes: an information acquisition module configured to acquire user feedback information on the target segment; and an adjustment module configured to adjust model parameters of the large model based on the feedback information.

[0126] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0127] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0128] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0129] Multiple components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a file processing method. For example, in some embodiments, the file processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the file processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the file processing method in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A file processing method, comprising: Extracting browsing information corresponding to the browsing attribute from the file to be processed; Extracting content information from the file to be processed; splicing the browsing information and the content information to obtain a target prompt word; The file to be processed and the target prompt word are input into the large model, and the target segment of the file to be processed is obtained as output.

2. The method according to claim 1, wherein: The step of extracting browsing information corresponding to the browsing attribute from the file to be processed includes: Extracting browsing data corresponding to the browsing attribute from the to-be-processed file, wherein the browsing data includes: real-time traffic data and / or real-time comment data; The browsing data is converted into structured data in a preset format.

3. The method according to claim 2, wherein: The content information includes picture information; and The extracting content information from the to-be-processed file includes: Segmenting the picture data of the file to be processed to obtain a plurality of segmented picture pictures; Performing image analysis on the lens picture to determine key frames in the lens picture; The picture information is extracted from the key frame, wherein the picture information includes at least one of the following: picture brightness information, picture scene information, and picture area information.

4. The method according to claim 3, wherein: The content information also includes audio information; and The extracting content information from the to-be-processed file further includes: Audio information is extracted from the audio data of the file to be processed, wherein the audio information includes at least one of the following: volume information, spectrum information, and rhythm information.

5. The method according to claim 4, wherein: The content information also includes text information; and The extracting content information from the to-be-processed file further includes: Natural language processing is performed on the text content of the to-be-processed file to determine text information corresponding to the text content, wherein the text information includes at least one of the following: keywords, sentiment words, and target paragraph information.

6. The method according to claim 5, further comprising: fusing the image information, the audio information, and the text information to obtain a fused first multimodal feature; as well as The step of combining the browsing information and the content information to obtain a target prompt word includes: The structured data and the first multimodal feature are concatenated to obtain the target prompt word.

7. The method according to claim 5, wherein: The files to be processed include music files; and The extracting content information from the to-be-processed file further includes: determining a key segment from the music file based on the frequency spectrum information; The target audio information of the key segment is obtained.

8. The method according to claim 7, wherein: The determining a key segment from the music file based on the frequency spectrum information comprises: Determine a plurality of peak points and trough points based on the frequency spectrum information, and calculate the difference between the peak points and the trough points; Determine a target peak point and a target trough point where the difference is greater than a preset threshold; The music segment corresponding to the target peak point and the target trough point is determined as the key segment.

9. The method according to claim 7, further comprising: The key segment is determined from the music file based on the user's behavioral characteristics.

10. The method according to claim 7, further comprising: fusing the picture information, the target audio information, and the text information to obtain a fused second multimodal feature; as well as The step of combining the browsing information and the content information to obtain a target prompt word includes: The structured data and the second multimodal feature are concatenated to obtain the target prompt word.

11. The method according to claim 5, wherein: The files to be processed include video files; and The extracting content information from the to-be-processed file further includes: Dividing the text content according to preset time levels to obtain multiple sub-texts; The sub-text is scored to generate score information of the sub-text.

12. The method according to claim 11, further comprising: fusing the picture information, the audio information, the text information, and the score information to obtain a fused third multimodal feature; as well as The step of combining the browsing information and the content information to obtain a target prompt word includes: The structured data and the third multimodal feature are concatenated to obtain the target prompt word.

13. The method according to claim 1, further comprising: The content integrity of the target segment is detected, and the interval value of the time interval of the target segment is adjusted based on the detection result.

14. The method according to claim 1, further comprising: Obtaining user feedback information on the target segment; Based on the feedback information, model parameters of the large model are adjusted.

15. A file processing device, comprising: A browsing information extraction module is configured to extract browsing information corresponding to the browsing attribute from the file to be processed; A content information extraction module, configured to extract content information from the file to be processed; A splicing module is configured to splice the browsing information and the content information to obtain a target prompt word; The output module is configured to input the to-be-processed file and the target prompt word into the large model, and output the target segment of the to-be-processed file.

16. The device according to claim 15, wherein: The browsing information extraction module is further configured to: Extracting browsing data corresponding to the browsing attribute from the to-be-processed file, wherein the browsing data includes: real-time traffic data and / or real-time comment data; The browsing data is converted into structured data in a preset format.

17. The device according to claim 16, wherein: The content information includes picture information; and The content information extraction module is further configured to: Segmenting the picture data of the file to be processed to obtain a plurality of segmented picture pictures; Performing image analysis on the lens picture to determine key frames in the lens picture; The picture information is extracted from the key frame, wherein the picture information includes at least one of the following: picture brightness information, picture scene information, and picture area information.

18. The device according to claim 17, wherein: The content information also includes audio information; and The content information extraction module is further configured to: Audio information is extracted from the audio data of the file to be processed, wherein the audio information includes at least one of the following: volume information, spectrum information, and rhythm information.

19. The device according to claim 18, wherein: The content information also includes text information; and The content information extraction module is further configured to: Natural language processing is performed on the text content of the to-be-processed file to determine text information corresponding to the text content, wherein the text information includes at least one of the following: keywords, sentiment words, and target paragraph information.

20. The apparatus according to claim 19, further comprising: a first fusion module configured to fuse the image information, the audio information, and the text information to obtain a fused first multimodal feature; and The splicing module is further configured to: The structured data and the first multimodal feature are concatenated to obtain the target prompt word.

21. The device according to claim 19, wherein The files to be processed include music files; and The content information extraction module comprises: a segment determination submodule, configured to determine a key segment from the music file based on the spectrum information; The information acquisition submodule is configured to acquire target audio information of the key segment.

22. The device according to claim 21, wherein The fragment determination submodule is further configured to: Determine a plurality of peak points and trough points based on the frequency spectrum information, and calculate the difference between the peak points and the trough points; Determine a target peak point and a target trough point where the difference is greater than a preset threshold; The music segment corresponding to the target peak point and the target trough point is determined as the key segment.

23. The apparatus according to claim 21, further comprising: The segment determination module is configured to determine the key segment from the music file based on the user's behavioral characteristics.

24. The apparatus according to claim 21, further comprising: a second fusion module configured to fuse the picture information, the target audio information and the text information to obtain a fused second multimodal feature; and The splicing module is further configured to: The structured data and the second multimodal feature are concatenated to obtain the target prompt word.

25. The device according to claim 19, wherein The files to be processed include video files; and The content information extraction module is further configured to: Dividing the text content according to preset time levels to obtain multiple sub-texts; The sub-text is scored to generate score information of the sub-text.

26. The apparatus according to claim 25, further comprising: a third fusion module configured to fuse the picture information, the audio information, the text information and the score information to obtain a fused third multimodal feature; and The splicing module is further configured to: The structured data and the third multimodal feature are concatenated to obtain the target prompt word.

27. The apparatus of claim 15, further comprising: The detection module is configured to detect the content integrity of the target segment and adjust the interval value of the time interval of the target segment based on the detection result.

28. The apparatus of claim 15, further comprising: An information acquisition module, configured to acquire user feedback information on the target segment; The adjustment module is configured to adjust the model parameters of the large model based on the feedback information.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method of any one of claims 1-14.

31. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images