Processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a processing method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: responding to a target text and a target video input by a user, generating a plurality of pieces of text information associated with the target text and the target video, determining similarity data of video frames in the target video and the text information, the video frames are screened based on the similarity data to obtain at least one target video frame, and the reply information is determined based on the target video frame and the target text, so that the video content processing capability of the intelligent model is improved, and the user experience is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the widespread adoption of the internet and mobile devices, video content has become a primary medium for information dissemination. The application scenarios for video content are increasingly diverse, placing higher demands on intelligent video understanding technology. In recent years, with the development of AI technology, breakthroughs have been achieved in text and image processing based on intelligent models. However, there are still shortcomings in video processing based on intelligent models.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, this disclosure provides a processing method, apparatus, electronic device and storage medium.

[0005] According to a first aspect of the present disclosure, a processing method is provided, comprising: In response to user input of target text and target video, generate multiple text messages associated with the target text and target video; Determine the similarity data between video frames and text information in the target video; Video frames are filtered based on similarity data to obtain at least one target video frame; The response information is determined based on the target video frame and the target text.

[0006] In one embodiment of this disclosure, the generation of target text and multiple text information associated with the target video includes: The target video is sampled to obtain at least one first sampled frame; The first sampled frame and the target text are input into a pre-trained multimodal large model to obtain multiple text information.

[0007] In one embodiment of this disclosure, sampling the target video to obtain at least one first sampled frame includes: The target video is sampled using an average sampling method to obtain at least one first sampled frame.

[0008] In one embodiment of this disclosure, determining the similarity data between video frames and text information in a target video includes: The target video is sampled to obtain at least one second sampled frame; The second sampled frame and text information are input into a pre-trained image-text similarity calculation model to obtain similarity data between the second sampled frame and the text information.

[0009] In one embodiment of this disclosure, the method further includes: A similarity curve is generated based on the similarity data corresponding to the second sampled frame.

[0010] In one embodiment of this disclosure, video frames are filtered based on similarity data to obtain at least one target video frame, including: The similarity curve is smoothed and denoised to obtain the processed similarity curve; Segments with similarity scores above a preset similarity threshold are identified as target segments. At least one third sampling frame is obtained by sampling the target segment based on the first sampling frequency; Based on the second sampling frequency, the segments other than the target segment are sampled to obtain at least one fourth sampling frame, where the first sampling frequency is higher than the second sampling frequency; The target video frame is determined based on the third and fourth sampled frames.

[0011] In one embodiment of this disclosure, determining the response information corresponding to the target text based on the target video frame and the target text includes: The target video frame and target text are input into a pre-trained multimodal large model to obtain response information corresponding to the target text.

[0012] According to another aspect of this disclosure, a processing apparatus is provided, comprising: The first generation module is used to generate multiple text information associated with the target text and target video in response to user input of target text and target video; The first determining module is used to determine the similarity data between video frames and text information in the target video; The filtering module is used to filter video frames based on similarity data to obtain at least one target video frame; The second determining module is used to determine the response information based on the target video frame and the target text.

[0013] In one embodiment of this disclosure, the first generation module includes: The first sampling unit is used to sample the target video to obtain at least one first sampling frame; The first input unit is used to input the first sampled frame and the target text into the pre-trained multimodal large model to obtain multiple text information.

[0014] In one embodiment of this disclosure, the first sampling unit includes: The sampling subunit is used to sample the target video based on average sampling to obtain at least one first sampled frame.

[0015] In one embodiment of this disclosure, the first determining module includes: The second sampling unit is used to sample the target video to obtain at least one second sampling frame; The second input unit is used to input the second sampled frame and text information into the pre-trained image-text similarity calculation model to obtain the similarity data between the second sampled frame and the text information.

[0016] In one embodiment of this disclosure, the apparatus further includes: The second generation module is used to generate a similarity curve based on the similarity data corresponding to the second sampling frame.

[0017] In one embodiment of this disclosure, the filtering module includes: The processing unit is used to smooth and reduce noise on the similarity curve based on the Gaussian kernel function to obtain the processed similarity curve. The first determining unit is used to determine the segments in the similarity curve that are higher than a preset similarity threshold as target segments; The third sampling unit is used to sample the target segment based on the first sampling frequency to obtain at least one third sampling frame; The fourth sampling unit is used to sample segments other than the target segment based on the second sampling frequency to obtain at least one fourth sampling frame, wherein the first sampling frequency is higher than the second sampling frequency; The second determining unit is used to determine the target video frame based on the third and fourth sampled frames.

[0018] In one embodiment of this disclosure, the second determining module includes: The third input unit is used to input the target video frame and the target text into the pre-trained multimodal large model to obtain the response information corresponding to the target text.

[0019] According to a third aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement any of the processing methods described in the first aspect above.

[0020] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a terminal, enables the terminal to perform any of the processing methods described in the first aspect.

[0021] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the processing methods described in the first aspect.

[0022] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: This disclosure responds to user input of target text and target video, generates multiple text information associated with the target text and target video, determines the similarity data between video frames and text information in the target video, filters video frames based on the similarity data to obtain at least one target video frame, and determines the response information based on the target video frame and target text, thereby improving the intelligent model's ability to process video content and enhancing the user experience.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0025] Figure 1 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 1 .

[0026] Figure 2 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 2 .

[0027] Figure 3 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 3 .

[0028] Figure 4 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 4 .

[0029] Figure 5 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 5 .

[0030] Figure 6 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 6 .

[0031] Figure 7 This is a processing apparatus block shown according to an exemplary embodiment of the present disclosure. Figure 1 .

[0032] Figure 8 This is a block diagram illustrating an apparatus for the above-described processing method or the above-described processing method, according to some embodiments of the present disclosure. Detailed Implementation

[0033] Exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0034] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0035] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0036] Figure 1 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 1 The processing method can be applied to terminals, including but not limited to mobile phones, computers, smart wearable devices, smart home devices, smart cars, VR devices, and AR devices. It can also be applied to server-side devices such as local servers and cloud servers, which can be deployed on a single computer or a computer cluster consisting of multiple computers.

[0037] like Figure 1 As shown, it includes the following steps.

[0038] S110, in response to user input of target text and target video, generates multiple text information associated with the target text and target video.

[0039] In some embodiments, the target text may include information related to the target video sent by the user. It should be noted that the user can send voice information, action information, and text information. After the terminal receives the voice information and / or action information, it can convert the voice information and / or action information into text information. The text information in this disclosure may include questions about the content of the target video, information for optimizing the target video, or instructions for extracting video segments from the target video. For example, the target text may include: which team won the match between the two teams in the video, beautifying a task in the video, or extracting a video segment containing a specified item, etc.

[0040] In some embodiments, the target video may include the video that the user needs to interact with. The target video can be any video provided by the user, such as a short video, a live stream, or a long video. It should be noted that when the target video is a live stream, the target text may include instructions for interacting with the previously streamed video, as well as instructions for interacting with the upcoming live stream.

[0041] In some embodiments, the aforementioned speech information, action information, and text information can be converted into target text based on a pre-trained intelligent model. The pre-trained intelligent model may include an audio-to-text (ASR) model, an image-to-text and visual question-answering model, and an integrated audio and end-to-end solution model.

[0042] In some embodiments, the text information may include relevant information for expanding the information contained in the target text. For example, if the target text is "Which team won the game in the video just now?", the text information may include information such as "What kind of game was the game in the video?", "What are the differences between the different teams?", and "What are the characteristics of winning the game?".

[0043] S120, determine the similarity data between video frames and text information in the target video.

[0044] In some embodiments, a video frame may include any image frame that makes up the target video.

[0045] In some embodiments, the similarity data between video frames and text information may include the similarity data between each video frame and at least one piece of text information. It should be noted that the target video may include multiple video frames, and the similarity data between each video frame and each piece of text information can be determined.

[0046] In some embodiments, by determining multiple text information associated with the target video and the target text, and then determining similarity data between the target video frames and the text information, more important video frames in the target video can be retained based on the similarity data.

[0047] S130, Based on similarity data, video frames are filtered to obtain at least one target video frame.

[0048] In some embodiments, filtering video frames based on similarity data may include identifying video frames with higher similarity data as target video frames for retention. It should be noted that any video frame whose similarity data with any text information exceeds a preset threshold or preset range can be retained as a target video frame.

[0049] In some embodiments, a smaller selection of video frames is made based on similarity data to obtain target video frames. Then, response information is determined based on the video frames and target text. This not only reduces the amount of data to be processed and improves processing efficiency, but also avoids the impact of invalid data in the target data on the accuracy of the response information.

[0050] S140, determine the response information based on the target video frame and the target text.

[0051] In some embodiments, the response information may include information for responding to the target text, and the response information may include text information, video clips, and voice information. This disclosure does not limit the type of response information; the type of response information may be determined based on the textual meaning of the target text.

[0052] In some embodiments, the response information can be determined based on a pre-trained intelligent model, the target video frame, and the target text.

[0053] This disclosure responds to user input of target text and target video, generates multiple text information associated with the target text and target video, determines the similarity data between video frames and text information in the target video, filters video frames based on the similarity data to obtain at least one target video frame, and determines the response information based on the target video frame and target text, thereby improving the intelligent model's ability to process video content and enhancing the user experience.

[0054] Figure 2 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 2 . Figure 2 Steps S230 to S250 correspond to steps S120 to S140 and will not be repeated here. Figure 2 As shown, in Figure 1 In addition to the implementation process shown, the following steps are also included: S210, sample the target video to obtain at least one first sampled frame.

[0055] In some embodiments, sampling the target video may include acquiring a small number of sample frames from multiple video frames contained in the target video. It should be noted that the target video may be sampled using random sampling, adaptive sampling, hierarchical sampling, temporal sampling, and spatial sampling methods in this disclosure. Random sampling may include randomly acquiring a certain number or proportion of frames from all sample frames of the target video. Adaptive sampling may include using models or rules to determine the "information content" or "representativeness" of each frame and dynamically adjusting the sampling frequency. Hierarchical sampling may include first dividing the video / data into several layers, and then sampling within each layer. Temporal sampling may include selecting frames or sampling points according to the timeline. Spatial sampling may include selecting a local region or downsampling from a high-resolution image according to spatial rules for a single frame image.

[0056] In some embodiments, sampling the target video may include sampling the target video based on average sampling to obtain at least one first sampled frame. It should be noted that, unlike slow sampling and fast sampling, uniform sampling may include sampling the target video at fixed intervals to ensure that the sampled frames are evenly distributed over time.

[0057] In some embodiments, uniform sampling can be achieved using either fixed-interval sampling or time-uniform sampling. It should be noted that each video frame corresponds to a time point, and with time-uniform sampling, the interval between any two adjacent first sampled frames can be guaranteed to be the same.

[0058] S220 inputs the first sampled frame and the target text into the pre-trained multimodal large model to obtain multiple text information.

[0059] In some embodiments, the multimodal large model can be used to understand and analyze video and text, but its performance is poor when processing low-quality video. The multimodal large model is based on a large language model and incorporates multimodal inputs such as images, audio, and video through modal encoders and modal interfaces to achieve a unified model framework for understanding, reasoning, and generating multimodal information.

[0060] In some embodiments, when inputting the target text and the first sampling frame into the multimodal large model, the multimodal large model may also be input with prompt words related to the generation of multiple text information.

[0061] In some embodiments, the method in this disclosure may further include training a large multimodal model through multimodal pre-training, instruction fine-tuning, and alignment fine-tuning. It should be noted that multimodal pre-training enables the alignment of representations on large-scale image-text peer-to-peer data and the learning of visual token reading. Instruction fine-tuning can improve compliance and generalization with multi-task instruction data. Alignment fine-tuning can improve reliability and security by leveraging human preferences / feedback.

[0062] In this embodiment of the disclosure, at least one first sampling frame is obtained by sampling the target video. The first sampling frame and the target text are then input into a pre-trained multimodal large model to obtain multiple text information. Sampling reduces the amount of data input to the multimodal large model and improves processing efficiency.

[0063] Figure 3 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 3 . Figure 3 Steps S310, S340, and S350 correspond to steps S110, S130, and S140, and will not be repeated here. Figure 3 As shown, in Figure 1 In addition to the implementation process shown, the following steps are also included: S320, sample the target video to obtain at least one second sampled frame.

[0064] In some embodiments, the sampling method for the target video can be the same as that in the above embodiments.

[0065] In some embodiments, the target video may be densely sampled to obtain at least one second sampled frame. Dense sampling may include a strategy of selecting data with high density in the temporal, spatial, or feature dimensions, with the aim of capturing information as comprehensively and meticulously as possible and avoiding omission of key content. It is the opposite of "sparse sampling" or "uniform sampling" and is often used in scenarios where high detail is required, changes are frequent, or the distribution of key information is uncertain.

[0066] S330, input the second sampled frame and text information into the pre-trained image-text similarity calculation model to obtain the similarity data between the second sampled frame and the text information.

[0067] In some embodiments, the image-text similarity calculation model maps images and text to the same semantic space through contrastive learning, and uses InfoNCE loss to learn pairing relationships in a large batch of negative examples; during inference, the matching score can be obtained by taking the cosine similarity of image and text features, which naturally supports zero-shot classification and cross-modal retrieval.

[0068] In some embodiments, the similarity data may include data that reflects the degree of similarity between the second sampled frame and the text information. The similarity data may include relevant data represented numerically and graphically.

[0069] In some embodiments, when there are multiple pieces of text information, the similarity data between each piece of text information and any one of the second sampling frames can be obtained separately. This yields the similarity data between each second sampling frame and any one of the multiple pieces of text information.

[0070] In this embodiment of the disclosure, at least one second sampled frame is obtained by sampling the target video. The second sampled frame and text information are then input into a pre-trained image-text similarity calculation model to obtain similarity data between the second sampled frame and the text information. The sampling frames with higher importance in the target video are determined by using the similarity data, which reduces the amount of data to be processed and improves processing efficiency.

[0071] Figure 4 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 4 . Figure 4 Steps S410 to S430 and S460 correspond to steps S310, S320, S330 and S350, and will not be repeated here. Figure 4 As shown, in Figure 3 In addition to the implementation process shown, the following steps are also included: S440, Generate a similarity curve based on the similarity data corresponding to the second sampled frame.

[0072] In some embodiments, the similarity curve may include a curve representing the similarity between the second sampled frame and the text information. This curve can represent the trend of similarity between the second sampled frame and the text information and the change in variables.

[0073] In some embodiments, since the second sampled frame has a time identifier, the second sampled frame can be sorted based on the time identifier, and then a similarity curve can be generated based on the similarity data corresponding to the second sampled frame.

[0074] In some embodiments, the similarity data corresponding to the second sampling frame may include the average of the similarity data of multiple text information corresponding to the second sampling frame. For example, if the similarity data of the multiple text information corresponding to the second sampling frame are 0.8, 0.2, 0.6, and 0.4 respectively, then the similarity data corresponding to the second sampling frame is (0.8+0.2+0.6+0.4) / 4=0.5.

[0075] In some embodiments, after obtaining the similarity curves, overlapping portions of the similarity curves can be merged. For example, rule-based or threshold-based merging methods can be used to reduce, filter, or discard overlapping regions in the similarity curves. Alternatively, the similarity curves can be divided into time intervals / spatial regions; if two intervals overlap, they are merged into a larger interval, and the similarity within that interval is aggregated. Local fusion based on a sliding window can also be used to merge overlapping portions. By using a sliding window, multiple similarity regions within the window are locally fused.

[0076] S450: Based on the similarity curve, video frames are filtered to obtain at least one target video frame.

[0077] In this embodiment of the disclosure, a similarity curve is generated by the similarity data corresponding to the second sampling frame, which can clearly reflect the differences in the similarity data corresponding to different second sampling frames, making it convenient to filter the second sampling frames based on the similarity curve.

[0078] Figure 5 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 5 . Figure 5 Steps S510, S520, and S580 correspond to steps S110, S120, and S140, and will not be repeated here. Figure 5 As shown, in Figure 1 In addition to the implementation process shown, the following steps are also included: S530 uses a Gaussian kernel function to smooth and denoise the similarity curve, resulting in the processed similarity curve.

[0079] In some embodiments, the Gaussian kernel function can be a one-dimensional discrete Gaussian filter function, which can perform weighted averaging on the similarity curve. The one-dimensional discrete Gaussian filter function may include:

[0080] in, To smooth out the operation, in The value on. Among them For similarity data, This is the second sampling frame. For the radius of the domain, This is the offset. as well as All settings can be customized by the user. Indicates in Location similarity data It can be the standard deviation of the Gaussian function. It can be a normalization coefficient.

[0081] For example, this disclosure can Set to 1, Set it to 4 to smooth and reduce noise in the similarity curve.

[0082] In some embodiments, the method in this disclosure may further include: smoothing and denoising the similarity curve based on methods such as moving average, spline smoothing, and low-pass filtering.

[0083] In some embodiments, the one-dimensional discrete Gaussian filter function can preserve trends in addition to smoothing similarity curves.

[0084] S540, identify segments in the similarity curve that are higher than the preset similarity threshold as target segments.

[0085] In some embodiments, the similarity threshold can be set by the user.

[0086] In some embodiments, the similarity data corresponding to different second sampling frames are different. By dividing the second sampling frames into target segments and non-target segments, the target segments and non-target segments can be processed separately, thereby improving the processing accuracy.

[0087] S550, sample the target segment based on the first sampling frequency to obtain at least one third sampling frame.

[0088] S560, sampling segments other than the target segment is performed based on the second sampling frequency to obtain at least one fourth sampling frame, wherein the first sampling frequency is higher than the second sampling frequency.

[0089] In some embodiments of this disclosure, target segments can be sampled based on fast sampling, and non-target segments can be sampled based on slow sampling. Fast sampling can involve sampling the signal or data at a higher frequency, i.e., short sampling intervals and dense sampling points. Slow sampling can involve sampling the signal or data at a lower frequency, i.e., long sampling intervals and sparse sampling points.

[0090] In some embodiments, a first sampling frequency higher than a second sampling frequency may be included in video segments of the same length, and the number of video frames sampled based on the first sampling frequency is greater than the number of video frames sampled at the second sampling frequency.

[0091] S570 determines the target video frame based on the third and fourth sampled frames.

[0092] In some embodiments, determining the target video frame based on the third sampled frame and the fourth sampled frame may include synthesizing the third sampled frame and the fourth sampled frame to obtain the target video frame.

[0093] In this embodiment, a Gaussian kernel function is used to smooth and reduce noise using a similarity curve. Then, the similarity curve is divided into target segments and non-target segments. The target segments and non-target segments are sampled based on different sampling frequencies. This ensures that more video frames are obtained for segments with higher similarity, while fewer video frames are obtained for segments with lower similarity. This ensures that video frames with high relevance to the target text are fully sampled while avoiding information loss.

[0094] Figure 6 This is a flowchart of a processing method according to an exemplary embodiment of the present disclosure. Figure 6 . Figure 6 Steps S610 to S630 correspond to steps S110 to S130 and will not be repeated here. Figure 6 As shown, in Figure 1 In addition to the implementation process shown, the following steps are also included: S640 inputs the target video frame and target text into a pre-trained multimodal large model to obtain response information corresponding to the target text.

[0095] In some embodiments, the multimodal large model in this embodiment may be the same as the multimodal large model in the above embodiments, and will not be repeated here.

[0096] This disclosure responds to user input of target text and target video, generates multiple text information associated with the target text and target video, determines the similarity data between video frames and text information in the target video, filters video frames based on the similarity data to obtain at least one target video frame, and determines the response information based on the target video frame and target text, thereby improving the intelligent model's ability to process video content and enhancing the user experience.

[0097] To provide a detailed description of this disclosure, an exemplary embodiment is also provided. The processing method in this disclosure is configured on a terminal. The terminal responds to user input of a relevant instruction by parsing the instruction. If the instruction is a non-text instruction, it is converted into a text instruction, which can be target text. The text instruction is then input into an intelligent model capable of understanding text. If the intelligent model determines that the text is used for video processing, the target video indicated by the user is obtained. The target video can be a video stored on the terminal, a video that has already been played, or a live video. After the terminal obtains the target video, it can perform uniform sampling on the target video. The sampled video frames and the text instruction are then input into a pre-trained multimodal large-scale model. Simultaneously with inputting into the pre-trained multimodal large-scale model, the terminal can also input relevant prompt words generated based on the text instruction, wherein the relevant prompt words can be generated by a large language model.

[0098] After inputting video frames and text commands into a pre-trained multimodal large model, the output text information of the multimodal large model can be obtained. The text information may include extended text information related to the text commands in the target video. The terminal can then perform intensive sampling of the target video to obtain second sampled frames. After obtaining the second sampled frames, the terminal can input the second sampled frames and text information into a pre-trained image-text similarity calculation model to obtain similarity data output by the pre-trained image-text similarity calculation model. It should be noted that since there may be multiple pieces of text information, the similarity data between each second sampled frame and each piece of text information can be calculated separately. Then, the average of these multiple similarity data is used to obtain the final similarity data.

[0099] After obtaining the final similarity data, a similarity curve can be generated based on the aforementioned similarity data and the time corresponding to the second sampling frame. Then, a Gaussian kernel function is used to preprocess the similarity curve, which may include smoothing, noise reduction, and merging of overlapping parts. After preprocessing the similarity curve, a threshold method can be used to divide the similarity curve into high-interest segments and non-high-interest segments. High-interest segments are those with relatively high similarity. After dividing into high-interest segments, fast sampling can be performed on the high-interest segments, and slow sampling on the non-high-interest segments to obtain multiple sampling frames. After obtaining multiple sampling frames, these sampling frames and the target text can be input into a pre-trained multimodal large-scale model, which outputs a text response. Upon receiving the text response at the terminal, the text response can be converted into speech or image information.

[0100] It should be noted that this disclosure can also use the Gaussian kernel function to divide the curve into high-interest segments and non-high-interest segments. The Gaussian kernel function can be:

[0101] In Set to 1, When set to 4, both high-interest segments and non-high-interest segments can be obtained.

[0102] Based on the same inventive concept, this disclosure also provides a processing apparatus, as shown in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this apparatus embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be elaborated further.

[0103] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0104] Figure 7 This is a processing apparatus block shown according to an exemplary embodiment of the present disclosure. Figure 1 .

[0105] Reference Figure 7 The device 700 includes: a first generation module 710, a first determination module 720, a filtering module 730, and a second determination module 740.

[0106] The first generation module 710 is used to generate multiple text information associated with the target text and the target video in response to user input of target text and target video; The first determining module 720 is used to determine the similarity data between video frames and text information in the target video; The filtering module 730 is used to filter video frames based on similarity data to obtain at least one target video frame; The second determining module 740 is used to determine the response information based on the target video frame and the target text.

[0107] This disclosure responds to user input of target text and target video, generates multiple text information associated with the target text and target video, determines the similarity data between video frames and text information in the target video, filters video frames based on the similarity data to obtain at least one target video frame, and determines the response information based on the target video frame and target text, thereby improving the intelligent model's ability to process video content and enhancing the user experience.

[0108] In one embodiment of this disclosure, the first generation module 710 includes: The first sampling unit is used to sample the target video to obtain at least one first sampling frame; The first input unit is used to input the first sampled frame and the target text into the pre-trained multimodal large model to obtain multiple text information.

[0109] In one embodiment of this disclosure, the first sampling unit includes: The sampling subunit is used to sample the target video based on average sampling to obtain at least one first sampled frame.

[0110] In one embodiment of this disclosure, the first determining module 720 includes: The second sampling unit is used to sample the target video to obtain at least one second sampling frame; The second input unit is used to input the second sampled frame and text information into the pre-trained image-text similarity calculation model to obtain the similarity data between the second sampled frame and the text information.

[0111] In one embodiment of this disclosure, the apparatus further includes: The second generation module is used to generate a similarity curve based on the similarity data corresponding to the second sampling frame.

[0112] In one embodiment of this disclosure, the screening module 730 includes: The processing unit is used to smooth and reduce noise on the similarity curve based on the Gaussian kernel function to obtain the processed similarity curve. The first determining unit is used to determine the segments in the similarity curve that are higher than a preset similarity threshold as target segments; The third sampling unit is used to sample the target segment based on the first sampling frequency to obtain at least one third sampling frame; The fourth sampling unit is used to sample segments other than the target segment based on the second sampling frequency to obtain at least one fourth sampling frame, wherein the first sampling frequency is higher than the second sampling frequency; The second determining unit is used to determine the target video frame based on the third and fourth sampled frames.

[0113] In one embodiment of this disclosure, the second determining module 740 includes: The third input unit is used to input the target video frame and the target text into the pre-trained multimodal large model to obtain the response information corresponding to the target text.

[0114] This disclosure responds to user input of target text and target video, generates multiple text information associated with the target text and target video, determines the similarity data between video frames and text information in the target video, filters video frames based on the similarity data to obtain at least one target video frame, and determines the response information based on the target video frame and target text, thereby improving the intelligent model's ability to process video content and enhancing the user experience.

[0115] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0116] Figure 8 This is a block diagram illustrating an apparatus for the above-described processing method or the above-described processing method, according to some embodiments of this disclosure. For example, apparatus 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0117] Reference Figure 8 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0118] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0119] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0120] The power supply component 906 provides power to the various components of the device 900. The power supply component 906 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 900.

[0121] Multimedia component 908 includes a screen that provides an output interface between device 900 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When device 900 is in an operating mode, such as shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0122] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice processing mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0123] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0124] Sensor assembly 914 includes one or more sensors for providing state assessments of various aspects of device 900. For example, sensor assembly 914 may detect the on / off state of device 900, the relative positioning of components such as the display and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0125] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency processing (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0126] In some embodiments of this disclosure, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0127] In some embodiments of this disclosure, a computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of the device 900 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0128] A computer-readable storage medium that, when instructions in the storage medium are executed by a terminal's processor, enables the terminal to perform an operation instruction execution method or a method for operation instruction execution.

[0129] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0130] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A processing method, characterized in that, include: In response to user input of target text and target video, generate multiple text information associated with the target text and target video; Determine the similarity data between video frames in the target video and the text information; Based on the similarity data, the video frames are filtered to obtain at least one target video frame; The response information is determined based on the target video frame and the target text.

2. The method according to claim 1, characterized in that, The generation of the target text and the multiple text information associated with the target video includes: The target video is sampled to obtain at least one first sampled frame; The first sampled frame and the target text are input into a pre-trained multimodal large model to obtain the multiple text information.

3. The method according to claim 2, characterized in that, The step of sampling the target video to obtain at least one first sampled frame includes: The target video is sampled using an average sampling method to obtain at least one first sampled frame.

4. The method according to claim 1, characterized in that, The step of determining the similarity data between video frames in the target video and the text information includes: The target video is sampled to obtain at least one second sampled frame; The second sampled frame and the text information are input into a pre-trained image-text similarity calculation model to obtain similarity data between the second sampled frame and the text information.

5. The method according to claim 4, characterized in that, The method further includes: A similarity curve is generated based on the similarity data corresponding to the second sampled frame.

6. The method according to claim 5, characterized in that, The step of filtering the video frames based on the similarity data to obtain at least one target video frame includes: The similarity curve is smoothed and denoised to obtain the processed similarity curve; The segments in the similarity curve that are higher than a preset similarity threshold are identified as target segments; At least one third sampling frame is obtained by sampling the target segment based on the first sampling frequency; Based on the second sampling frequency, segments other than the target segment are sampled to obtain at least one fourth sampling frame, wherein the first sampling frequency is higher than the second sampling frequency; The target video frame is determined based on the third and fourth sampled frames.

7. The method according to claim 1, characterized in that, The step of determining the response information corresponding to the target text based on the target video frame and the target text includes: The target video frame and the target text are input into a pre-trained multimodal large model to obtain response information corresponding to the target text.

8. A processing apparatus, characterized in that, include: The first generation module is used to generate multiple text information associated with the target text and the target video in response to user input of target text and target video; The first determining module is used to determine the similarity data between the video frames in the target video and the text information. A filtering module is used to filter the video frames based on the similarity data to obtain at least one target video frame; The second determining module is used to determine the response information based on the target video frame and the target text.

9. The apparatus according to claim 8, characterized in that, The first generation module includes: The first sampling unit is used to sample the target video to obtain at least one first sampling frame; The first input unit is used to input the first sampled frame and the target text into a pre-trained multimodal large model to obtain the multiple text information.

10. The apparatus according to claim 8, characterized in that, The first determining module includes: The second sampling unit is used to sample the target video to obtain at least one second sampling frame; The second input unit is used to input the second sampled frame and the text information into a pre-trained image-text similarity calculation model to obtain similarity data between the second sampled frame and the text information.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the processing method according to any one of claims 1 to 7.

12. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to perform the steps of a processing method according to any one of claims 1 to 7.

13. A computer program product, said computer program product comprising a computer program or computer instructions, characterized in that, The computer program or the computer instructions are loaded and executed by the processor to enable the computer to perform the steps of the processing method as described in any one of claims 1-7.

Images