Video processing method and device, computer equipment and storage medium

Abstract

The invention discloses a video processing method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of performing framing processing on a to-be-processed video to obtain a video frame group; inputting the plurality of video frame groups, the video description text and the video candidate box into a pre-trained saliency detection model to obtain a saliency video frame group; obtaining a candidate frame region corresponding to the significant video frame group, and obtaining a real labeling region corresponding to the to-be-processed video; calculating the prediction loss of the saliency detection model based on the candidate frame region and the real labeling region; iteratively updating the saliency detection model based on the prediction loss to obtain an updated saliency detection model; and obtaining a to-be-queried text, and inputting the to-be-queried text into the updated saliency detection model to obtain a video query result. The invention further relates to the technical field of block chains, and the to-be-queried text is stored in a block chain network. According to the invention, the positioning precision of the model on the video clip is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a video processing method, apparatus, computer equipment and storage medium. Background Art

[0002] Video clip retrieval refers to the process of finding the required video clips from a large amount of video data. On video sharing websites, online video websites and other platforms, users can enter keywords or descriptions to find film and television clips, variety show clips, etc. of interest, which helps users quickly locate the content they want to watch and improves user experience. Taking online insurance claims learning tutorial videos as an example, users can enter keywords such as "auto insurance claims process" or describe specific scenes to accurately find relevant teaching clips, thereby saving time and focusing on learning key points.

[0003] The traditional method of video clip retrieval in the industry is generally to manually label the video, write text content for each video clip, and then retrieve the corresponding video clip in the form of text retrieval. This method relies too much on manual operation, has low efficiency, and is easily affected by subjective factors. In addition, manual video annotation is time-consuming, labor-intensive, and costly.

[0004] With the continuous development of artificial intelligence technology, video clip retrieval has become more intelligent. At present, some image description models based on machine learning technology extract key frames from the video, describe the extracted key frames in text, and then use the retrieval model to use text-to-text retrieval to retrieve video clips. However, when processing videos with a high degree of direct correlation between frames, this method will be very inaccurate in locating video clips, and there will be conflicts in the retrieval of similar images, and the positioning is unclear, resulting in a poor user experience. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a video processing method, apparatus, computer equipment and storage medium to solve the technical problems in the prior art that, when processing videos with a high degree of direct correlation between frames, the positioning of video clips is very inaccurate, and conflicts may occur in the retrieval of similar images, the positioning is unclear, and the user experience is poor.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a video processing method, which adopts the following technical solution:

[0007] A video processing method, comprising:

[0008] Obtain the video to be processed, and perform frame processing on the video to be processed to obtain several video frame groups;

[0009] Obtaining video description text and video candidate frames, and inputting several video frame groups, video description text and video candidate frames into a pre-trained saliency detection model to obtain salient video frame groups;

[0010] Obtain the candidate frame area corresponding to the salient video frame group, and obtain the real marked area corresponding to the video to be processed;

[0011] Based on the candidate box area and the true labeled area, calculate the prediction loss of the saliency detection model;

[0012] Iteratively update the saliency detection model based on the prediction loss until the model is fitted, and obtain an updated saliency detection model;

[0013] The text to be queried is obtained, and the text to be queried is input into the updated saliency detection model to obtain the video query result.

[0014] Further, the step of obtaining video description text and video candidate frames, and inputting a plurality of video frame groups, video description text and video candidate frames into a pre-trained saliency detection model to obtain salient video frame groups specifically includes:

[0015] Extracting video features from a plurality of video frame groups through a pre-trained video feature extraction model to obtain feature vectors of a plurality of video frame groups;

[0016] Perform text segmentation on the video description text, and use the pre-trained text feature extraction model to extract text features from the text segmentation to obtain a text segmentation feature vector;

[0017] Perform feature vector conversion on the video candidate frame to obtain a feature vector of the video candidate frame;

[0018] Several video frame group feature vectors, text word segmentation feature vectors and video candidate box feature vectors are input into a pre-trained saliency detection model to obtain salient video frame groups.

[0019] Furthermore, the step of inputting several video frame group feature vectors, text word segmentation feature vectors and video candidate frame feature vectors into a pre-trained saliency detection model to obtain a salient video frame group specifically includes:

[0020] For each video frame group feature vector, the saliency matrix of each video frame group feature vector is constructed by combining the text segmentation feature vector and the video candidate frame feature vector to obtain several saliency matrices;

[0021] Based on the saliency matrix, calculate the saliency value of each video frame group;

[0022] The significance value is compared with a preset significance threshold, and a video frame group whose significance value is greater than the significance threshold is determined as a significant video frame group.

[0023] Furthermore, for each video frame group feature vector, the saliency matrix of each video frame group feature vector is constructed by combining the text segmentation feature vector and the video candidate frame feature vector to obtain several saliency matrices, specifically including:

[0024] Calculate the feature vector sum of the text word segmentation feature vector and the video candidate box feature vector;

[0025] The inner product of the feature vector of each video frame group and the sum of the feature vectors is calculated to obtain several saliency matrices.

[0026] Furthermore, the prediction loss is the intersection-over-union loss. Based on the candidate box area and the true labeled area, the steps of calculating the prediction loss of the saliency detection model specifically include:

[0027] Determine the intersection area of ​​the candidate box area and the real annotation area;

[0028] And determine the union area of ​​the candidate box area and the true annotation area;

[0029] Calculate the ratio of the intersection area to the union area to obtain the first area intersection-union ratio;

[0030] Calculate the difference between the intersection area and the union area to obtain the difference area;

[0031] Calculate the ratio of the difference area to the union area to obtain the second area intersection-union ratio;

[0032] Based on the IoU ratio of the first region and the IoU ratio of the second region, the prediction loss of the saliency detection model is obtained.

[0033] Furthermore, after obtaining the video description text and the video candidate frame, and inputting the plurality of video frame groups, the video description text and the video candidate frame into the pre-trained saliency detection model to obtain the salient video frame group, the method further includes:

[0034] For each salient video frame group, separately calculating the saliency value of each video frame in the salient video frame group;

[0035] Based on the saliency value of each video frame, calculating the average saliency value of each salient video frame group;

[0036] According to the average significance value of each significant video frame group, a corresponding weight item is configured for each significant video frame group;

[0037] Based on the weight item of each salient video frame group, each salient video frame group is weighted to obtain a weighted salient video frame group.

[0038] Furthermore, the saliency detection model is iteratively updated based on the prediction loss until the model is fitted to obtain an updated saliency detection model, specifically including:

[0039] Import the weighted salient video frame group into the saliency detection model for iterative training;

[0040] In the iterative model training, the prediction loss is back-propagated in the saliency detection model;

[0041] Based on the features of the weighted salient video frame group, the network parameters of the saliency detection model are continuously adjusted until the prediction loss of the saliency detection model is less than a preset loss threshold, thereby obtaining an updated saliency detection model.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a video processing device, which adopts the following technical solution:

[0043] A video processing device, comprising:

[0044] A frame processing module is used to obtain a video to be processed and perform frame processing on the video to be processed to obtain a plurality of video frame groups;

[0045] A saliency detection module is used to obtain video description text and video candidate frames, and input a plurality of video frame groups, video description text and video candidate frames into a pre-trained saliency detection model to obtain salient video frame groups;

[0046] A region recognition module is used to obtain the candidate frame region corresponding to the significant video frame group and obtain the real marked region corresponding to the video to be processed;

[0047] The loss prediction module is used to calculate the prediction loss of the saliency detection model based on the candidate box area and the true labeled area;

[0048] An iterative update module is used to iteratively update the saliency detection model based on the prediction loss until the model is fitted to obtain an updated saliency detection model;

[0049] The video query module is used to obtain the text to be queried, input the text to be queried into the updated saliency detection model, and obtain the video query result.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0051] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above-mentioned video processing methods when executing the computer-readable instructions.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0053] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video processing method described in any one of the above are implemented.

[0054] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0055] The present application discloses a video processing method, device, computer equipment and storage medium, which belongs to the field of artificial intelligence technology. The present application obtains video frame groups through frame processing, introduces a pre-trained saliency detection model, and effectively identifies and extracts salient video frame groups by combining video description text and video candidate frames. Then, by comparing the candidate frame area of ​​the salient video frame group with the real annotated area of ​​the video, the prediction loss is accurately calculated, and the saliency detection model is iteratively updated accordingly until the model is highly fitted, which significantly improves the positioning accuracy of the model for video clips. The present application solves the conflict problem in similar picture retrieval, realizes clear and accurate video content positioning, optimizes the user experience, and enables users to obtain more accurate and efficient query results when querying specific video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0058] Figure 2 A flow chart showing an embodiment of a video processing method according to the present application is shown;

[0059] Figure 3 A schematic structural diagram of an embodiment of a video processing device according to the present application is shown;

[0060] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0065] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0066] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0067] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0068] It should be noted that the video processing method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the video processing device is generally arranged in the server / terminal device.

[0069] It should be understood that Figure 1 The numbers of terminal devices, networks and servers in the embodiment are merely illustrative. The above system may have any number of terminal devices, networks and servers according to implementation requirements.

[0070] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for video processing according to the present application. The video processing method comprises the following steps:

[0071] S201, obtaining a video to be processed, and performing frame processing on the video to be processed to obtain a plurality of video frame groups.

[0072] Specifically, the video to be processed is the original video data provided by the user. The video to be processed contains a large number of frames, and the duration is usually divided in seconds. For ease of processing, the video is divided into several frame groups according to time, and each frame group contains several consecutive frames of images. For example, if 6 frames are taken per second, 0.5 seconds can be divided into a frame group, that is, each frame group consists of 3 frames. The purpose of framing is to reduce the processing complexity while maintaining the semantic integrity in time. Each video frame group can be regarded as a minimum analysis unit. In subsequent processing, these frame groups will be combined with the saliency detection model to calculate their relevance to the text description. The frame group division criteria need to be selected according to the specific application scenario. For example, a smaller time interval can be selected for short videos, while a larger interval can be selected for long videos to improve efficiency. Frame processing not only reduces the computational overhead, but also enables the model to capture dynamic information within the time period.

[0073] S202, obtaining video description text and video candidate frames, and inputting a plurality of video frame groups, video description text and video candidate frames into a pre-trained saliency detection model to obtain salient video frame groups.

[0074] Specifically, the video description text is a natural language query input by the user, which describes the specific content that needs to be found in the video. For example, the description can be "a person walking across a bridge" or "a vehicle entering a tunnel". The video candidate boxes are several time periods or areas pre-defined from the video, representing potential salient segments. These candidate boxes are usually generated by fixed time intervals or algorithms. The saliency detection model vectorizes the video frame groups, description text, and candidate box features, and calculates the relevance of each frame group with the text description. The input of the model includes multimodal features of the video, such as visual features, text features, and candidate box features. The model performs feature vector operations on the input features through inner products or other fusion methods, and then the model outputs the saliency score of each frame group to determine which frame groups are more consistent with the semantic intent of the description. The salient video frame groups are the frame groups with the highest scores, indicating that the model believes that these frame groups are most relevant to the description.

[0075] S203, obtaining a candidate frame area corresponding to the salient video frame group, and obtaining a real annotated area corresponding to the video to be processed.

[0076] Specifically, the candidate box area corresponding to the salient video frame group is the result of the prediction of the saliency detection model. Each salient frame group corresponds to a time range or spatial area, which represents the part that the model believes is highly relevant to the descriptive text. The true annotated area is manually provided annotation data, which indicates the area in the video that actually corresponds to the descriptive text. The true annotated area usually comes from the annotation team and is manually verified to ensure accuracy. The system evaluates the accuracy of the model's prediction results by matching the salient frame group and the true annotated area. The candidate box and the true annotated area need to be mapped to the same time or space scale for comparison, such as using timestamps, frame numbers, or location information of video clips.

[0077] S204: Calculate the prediction loss of the saliency detection model based on the candidate box area and the true labeled area.

[0078] Specifically, the prediction loss of the saliency detection model is used to measure the degree of match between the candidate box area and the true annotated area. Indicators such as GIoU (Generalized Intersection over Union) are usually used to calculate the similarity between the candidate box and the true box. GIoU can comprehensively consider the overlapping area of ​​the box, the overall bounding box area and the relative position, and provide a more robust matching measurement method. The smaller the prediction loss, the closer the output result of the saliency detection model is to the true annotated area. In addition, a weighted loss of the saliency score can be introduced, for example, considering the weight of the saliency value, so that the model pays more attention to the optimization of high saliency areas. Loss calculation not only helps evaluate model performance, but also provides direction for model parameter optimization. The final loss value is based on the average result of all candidate boxes, representing the overall prediction quality of the model.

[0079] S205, iteratively updating the saliency detection model based on the prediction loss until the model is fitted, thereby obtaining an updated saliency detection model.

[0080] Specifically, the update of the saliency detection model is achieved through standard optimization methods, such as the back-propagation algorithm based on gradient descent, with the prediction loss as the objective function. The parameters of the model are continuously adjusted according to the loss gradient to reduce the error. In each iteration, the saliency detection model recalculates the saliency value of the candidate box area and compares it with the real annotated area. The updated parameters make the prediction of the salient frame group closer to the true value. The iterative process can be combined with a dynamic learning rate or an early stopping mechanism to avoid overfitting or slow convergence of the model. The saliency detection model may be based on a deep learning framework (such as Transformer or a two-stream network), and multimodal features (video, text, time) are adjusted synchronously during the optimization process. After the iteration, the model can more accurately identify the salient frame group and output the optimized results.

[0081] S206, obtaining the text to be queried, and inputting the text to be queried into the updated saliency detection model to obtain a video query result.

[0082] Specifically, the text to be queried is the natural language requirement input by the user when using the saliency detection model, which describes the characteristics of the target video clip. The updated saliency detection model has been fully trained and can more efficiently understand the semantic association between text and video frame groups. When querying, the model first featurizes the text and fuses it with the video frame group features to calculate the saliency value, and outputs the video frame group that is most relevant to the description. These frame groups are sorted by saliency, and the frame segments with the highest saliency are returned as video query results. For example, if a user queries "a person running in the woods", the model will output the relevant video clip time range (such as 10 seconds to 15 seconds) or the corresponding video frame identifier. The video query results can be directly used in scenarios such as multimedia search and content recommendation to provide users with accurate text video matching services.

[0083] The solution of the present application calculates the prediction loss of the saliency detection model based on the candidate box area and the real annotated area; and it iteratively updates the saliency detection model based on the prediction loss until the model is fitted to obtain an updated saliency detection model. This operation accurately calculates the difference (prediction loss) between the video content position predicted by the model (candidate box area) and the real position (real annotated area), and continuously optimizes the model parameters accordingly, so that the model can more accurately identify and locate video content, especially for those video clips with a high correlation between frames and similar pictures. Through iterative updates, the model gradually learns how to distinguish different but similar pictures, thereby avoiding conflicts in retrieval and achieving clear and accurate positioning of video content.

[0084] In a specific embodiment of the present application, in order to help users quickly find specific teaching content, such as "Steps for filling out a car insurance claim form" or "Requirements for taking photos of an accident", etc. The solution of the present application first divides the insurance claim tutorial video into frames to generate a group of video frames, and inputs the pre-trained saliency detection model together with the video's description text (such as keywords or subtitles of the explanation content) and the candidate frame area (the area of ​​the screen containing key information). Through this model, it is possible to identify significant video frame groups in the video, such as screens containing important charts, filling demonstrations, or demonstration operations.

[0085] Then, the candidate frame areas corresponding to these salient frame groups are extracted, and the prediction error (i.e., prediction loss) of the saliency detection model is calculated using the real annotated areas of the video (such as clearly marked text or key operation locations in the picture) as a comparison. The saliency detection model is updated through iterative optimization based on the error, so that it can gradually more accurately identify the salient pictures that users pay attention to.

[0086] After the model optimization is completed, the user only needs to enter a query text such as "fill out a claim form" and the system will input it into the updated saliency detection model. The model will combine the salient frame groups in the video and the user input content to quickly locate the key clips explaining the steps of filling out the claim form and present it to the user as a search result.

[0087] This technical solution can significantly improve the query efficiency of insurance claims tutorials, allowing users to jump directly to the target segment without having to search for the required information frame by frame during the learning process. For example, when a user needs to know "how to upload accident photos", the system will accurately retrieve relevant images and provide explanation video clips, so that users can learn insurance claims operations efficiently.

[0088] In the above embodiment, the present application obtains video frame groups through frame processing, and introduces a pre-trained saliency detection model, which effectively identifies and extracts salient video frame groups by combining video description text and video candidate frames. Then, by comparing the candidate frame area of ​​the salient video frame group with the real annotated area of ​​the video, the prediction loss is accurately calculated, and the saliency detection model is iteratively updated accordingly until the model is highly fitted, which significantly improves the positioning accuracy of the model for video clips. The present application solves the conflict problem in similar picture retrieval, realizes clear and accurate video content positioning, optimizes the user experience, and enables users to obtain more accurate and efficient query results when querying specific video content.

[0089] Further, the step of obtaining video description text and video candidate frames, and inputting a plurality of video frame groups, video description text and video candidate frames into a pre-trained saliency detection model to obtain salient video frame groups specifically includes:

[0090] Extracting video features from a plurality of video frame groups through a pre-trained video feature extraction model to obtain feature vectors of a plurality of video frame groups;

[0091] Perform text segmentation on the video description text, and use the pre-trained text feature extraction model to extract text features from the text segmentation to obtain a text segmentation feature vector;

[0092] Perform feature vector conversion on the video candidate frame to obtain a feature vector of the video candidate frame;

[0093] Several video frame group feature vectors, text word segmentation feature vectors and video candidate box feature vectors are input into a pre-trained saliency detection model to obtain salient video frame groups.

[0094] In this embodiment, the feature extraction of several video frame groups is performed separately by the pre-trained video feature extraction model in order to convert the visual information of the original video into a computable feature representation. The feature vectors of these video frame groups contain the spatiotemporal information of the frame group, such as motion trajectory, object position and color, etc., which helps to capture the dynamic changes of the video content. After the text description text is segmented, it is input into the pre-trained text feature extraction model. The generated text segmentation feature vector contains the semantic information of the description text, such as features such as action, object and scene. The video candidate frame is a preset area for the video content, which is enabled to participate in the saliency calculation through feature vectorization. After the above three types of feature vectors are input into the saliency detection model, the model calculates the correlation score of each frame group with the description text through feature fusion, and selects the video frame group with higher saliency as output. In this way, the saliency detection model can effectively locate the most relevant fragments in the video with the text description and build a multimodal association between the video and the text.

[0095] Through the above steps, the saliency detection model achieves efficient video and text matching, can quickly and accurately identify video clips related to user needs, and improves the accuracy and efficiency of video search and content recommendation.

[0096] Furthermore, the step of inputting several video frame group feature vectors, text word segmentation feature vectors and video candidate frame feature vectors into a pre-trained saliency detection model to obtain a salient video frame group specifically includes:

[0097] For each video frame group feature vector, the saliency matrix of each video frame group feature vector is constructed by combining the text segmentation feature vector and the video candidate frame feature vector to obtain several saliency matrices;

[0098] Based on the saliency matrix, calculate the saliency value of each video frame group;

[0099] The significance value is compared with a preset significance threshold, and a video frame group whose significance value is greater than the significance threshold is determined as a significant video frame group.

[0100] In this embodiment, the system forms a saliency matrix by fusing the feature vector of the video frame group with the feature vector of the text segmentation and the feature vector of the video candidate frame, and can characterize the correlation between each video frame group and the text description. Specifically, the saliency matrix is ​​generated by vector inner product, weighted summation or other multimodal fusion methods, and the value of the matrix represents the similarity between the video frame group and the text description in a specific dimension. The saliency value is the result of normalizing or weighted summing the saliency matrix, representing the overall correlation. By comparing the saliency value of each video frame group with a preset threshold, a video frame group with a higher saliency value can be screened out. The setting of the saliency threshold is to balance the recall rate and the accuracy rate, and ensure that the screened frame group is highly correlated with the text description input by the user. The whole process realizes the efficient calculation of features in multimodality, and can quickly lock the most relevant frame group.

[0101] Through the above steps, the matching effect between the video frame group and the text description can be significantly improved, the video frame group with high significance can be screened out, the computational cost can be reduced, and the accuracy and robustness of the saliency detection model can be improved.

[0102] Furthermore, for each video frame group feature vector, the saliency matrix of each video frame group feature vector is constructed by combining the text segmentation feature vector and the video candidate frame feature vector to obtain several saliency matrices, specifically including:

[0103] Calculate the feature vector sum of the text word segmentation feature vector and the video candidate box feature vector;

[0104] The inner product of the feature vector of each video frame group and the sum of the feature vectors is calculated to obtain several saliency matrices.

[0105] In the present embodiment, first, the feature vector sum of the text segmentation feature vector and the video candidate frame feature vector is calculated. This step combines the semantics and spatiotemporal information of the text and the video candidate region to form a comprehensive feature vector. This comprehensive feature vector not only embodies the semantic content of the text description, but also reflects the spatiotemporal positioning information of the candidate frame. Next, each video frame group feature vector is subjected to an inner product operation with the above-mentioned comprehensive feature vector. This inner product calculation is to capture the correlation between the video frame group and the text description and the candidate frame, and to map the similarity of the three to the numerical value of the significance matrix. By calculating the frame group by frame, the significance matrix of each video frame group reflects the matching degree between the text segmentation and the candidate frame in the multidimensional feature space. This calculation method effectively integrates the multimodal feature information, so that the significance matrix can comprehensively characterize the significance of the video frame group.

[0106] Furthermore, the saliency matrix M i,j,k The expression is as follows:

[0107] M i,j,k =vi ·(t j +D k )

[0108] In the formula, v i is the feature vector of the i-th video frame group, t j is the feature vector of the jth text segmentation, D k is the feature vector of the kth video candidate box.

[0109] Through the above steps, the saliency matrix can be efficiently constructed to capture the correlation between video frame groups, text descriptions, and candidate boxes, thereby providing accurate multimodal feature fusion support for saliency detection and improving matching accuracy and analysis efficiency.

[0110] Furthermore, the prediction loss is the intersection-over-union loss. Based on the candidate box area and the true labeled area, the steps of calculating the prediction loss of the saliency detection model specifically include:

[0111] Determine the intersection area of ​​the candidate box area and the real annotation area;

[0112] And determine the union area of ​​the candidate box area and the true annotation area;

[0113] Calculate the ratio of the intersection area to the union area to obtain the first area intersection-union ratio;

[0114] Calculate the difference between the intersection area and the union area to obtain the difference area;

[0115] Calculate the ratio of the difference area to the union area to obtain the second area intersection-union ratio;

[0116] Based on the IoU ratio of the first region and the IoU ratio of the second region, the prediction loss of the saliency detection model is obtained.

[0117] In this embodiment, in the prediction loss calculation process of the saliency detection model, the intersection loss is the core indicator. First, the intersection area of ​​the candidate box area and the true annotation area is determined by geometric calculation. The intersection area represents the overlapping part of the model prediction result and the true value, which is a direct reflection of the accuracy. Next, the union area of ​​the two is calculated. The union area represents the overall area covered by the candidate box and the true annotation box, which is used to measure the integrity of the matching range. On this basis, the ratio of the intersection area to the union area is calculated to obtain the first area intersection ratio. The higher the first area intersection ratio, the higher the matching degree between the candidate box and the true annotation. For further optimization, the difference between the intersection area and the union area is calculated, and the ratio of the difference area to the union area is calculated to obtain the second area intersection ratio. The second area intersection ratio is used to quantify the impact of the unmatched part to ensure that the penalty for too large or too small boxes is more accurate. Finally, by combining the first area intersection ratio and the second area intersection ratio, the prediction loss value of the saliency detection model is obtained. The smaller the loss value, the better the model performance.

[0118] Furthermore, the loss function expression of the saliency detection model is as follows:

[0119]

[0120] Among them, A is the target area marked in the real data, that is, the real marked area, B is the target area predicted by the model, that is, the candidate box area, and C is the smallest area that can contain both A and B, that is, the union area. A, B, and C are all rectangular areas (not polygons). It is the ratio of the intersection area of ​​A and B to the union area of ​​A and B, that is, the first area intersection ratio, |C\(A∪B)| is the area remaining after subtracting the union area of ​​A and B from the area of ​​C, that is, the difference area. It is the ratio of the difference area to the union area of ​​A and B, that is, the second area intersection-union ratio. The larger the GIoU, the closer the model-predicted area B is to the real labeled area A, and the better the model effect. During training, in order to ensure that the loss function is always reduced, 1-GIoU can be used as the loss function expression of the saliency detection model.

[0121] Through the above steps, the intersection-over-union loss calculation method can comprehensively measure the matching degree between the candidate box area and the real labeled area, focusing on the accuracy of the overlapping parts and optimizing the impact of the unmatched parts. Ultimately, the robustness and prediction accuracy of the saliency detection model in various scenarios are improved.

[0122] Furthermore, after obtaining the video description text and the video candidate frame, and inputting the plurality of video frame groups, the video description text and the video candidate frame into the pre-trained saliency detection model to obtain the salient video frame group, the method further includes:

[0123] For each salient video frame group, separately calculating the saliency value of each video frame in the salient video frame group;

[0124] Based on the saliency value of each video frame, calculating the average saliency value of each salient video frame group;

[0125] According to the average significance value of each significant video frame group, a corresponding weight item is configured for each significant video frame group;

[0126] Based on the weight item of each salient video frame group, each salient video frame group is weighted to obtain a weighted salient video frame group.

[0127] In this embodiment, the refinement calculation of the significance value of the significant video frame group helps to improve the accuracy of significance detection. First, for each significant video frame group, the significance value of each video frame is calculated frame by frame, and these significance values ​​reflect the matching degree of each frame in the video frame group to the text description. Then, based on the significance values ​​of all frames in the video frame group, the average significance value of the frame group is calculated as the representative significance score of the frame group as a whole. Further, a corresponding weight item is configured for each frame group, and the weight item is set according to the importance of the frame group or a specific rule, such as the length of the frame group or other key features. By combining the significance value with the weight and weighting each significant video frame group, the influence of the important frame group can be amplified, and finally a weighted significant video frame group is generated. This process not only refines the significance analysis, but also enhances the preference for long-term related content through the weight mechanism, and improves the robustness of detection.

[0128] Through the above steps, through the refined calculation and weighted processing of the saliency values ​​of the frame groups, the saliency detection model can more accurately screen out the key video frame groups that meet the text description, thereby improving the accuracy of video content analysis and query results.

[0129] Furthermore, the saliency detection model is iteratively updated based on the prediction loss until the model is fitted to obtain an updated saliency detection model, specifically including:

[0130] Import the weighted salient video frame group into the saliency detection model for iterative training;

[0131] In the iterative model training, the prediction loss is back-propagated in the saliency detection model;

[0132] Based on the features of the weighted salient video frame group, the network parameters of the saliency detection model are continuously adjusted until the prediction loss of the saliency detection model is less than a preset loss threshold, thereby obtaining an updated saliency detection model.

[0133] In this embodiment, the saliency detection model is iteratively updated based on the prediction loss, with the purpose of optimizing the prediction ability of the model. First, the weighted saliency video frame group is input into the saliency detection model for iterative training, and the weighted saliency video frame group is prioritized according to its saliency score, so that the model pays more attention to those frame groups that are considered more relevant or important during the training process. Then, in each iteration, the model calculates the prediction loss and passes it back to the model through the back propagation algorithm. The back propagation guides the update of the model parameters by calculating the gradient of the loss function to the model parameters. Next, the model adjusts its network parameters based on the characteristics of the weighted saliency video frame group to reduce the prediction loss. This process continues until the prediction loss drops to a preset loss threshold, indicating that the model has converged and reached the optimal state. Finally, after the training is completed, an updated saliency detection model is obtained, which has higher prediction accuracy and better generalization ability, and can more effectively perform video clip retrieval and saliency detection in practical applications.

[0134] Through the above steps and the iterative update process, the saliency detection model can be continuously optimized in each round of training, ultimately improving the accuracy and stability of the model and ensuring that efficient and accurate saliency detection results can be provided in different scenarios.

[0135] In the above embodiment, the present application discloses a video processing method, which belongs to the field of artificial intelligence technology. The present application obtains video frame groups through frame processing, introduces a pre-trained saliency detection model, and combines video description text and video candidate frames to effectively identify and extract salient video frame groups. Then, by comparing the candidate frame area of ​​the salient video frame group with the real annotated area of ​​the video, the prediction loss is accurately calculated, and the saliency detection model is iteratively updated accordingly until the model is highly fitted, which significantly improves the positioning accuracy of the model for video clips. The present application solves the conflict problem in similar picture retrieval, realizes clear and accurate video content positioning, optimizes the user experience, and enables users to obtain more accurate and efficient query results when querying specific video content.

[0136] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0137] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned text information to be queried, the above-mentioned text information to be queried can also be stored in a node of a blockchain.

[0138] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0139] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0140] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0142] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0143] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a video processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0144] like Figure 3 As shown, the video processing device 300 described in this embodiment includes:

[0145] The frame processing module 301 is used to obtain the video to be processed and perform frame processing on the video to be processed to obtain a plurality of video frame groups;

[0146] A saliency detection module 302 is used to obtain video description text and video candidate frames, and input a plurality of video frame groups, video description text and video candidate frames into a pre-trained saliency detection model to obtain salient video frame groups;

[0147] The region identification module 303 is used to obtain the candidate frame region corresponding to the significant video frame group and obtain the real marked region corresponding to the video to be processed;

[0148] The loss prediction module 304 is used to calculate the prediction loss of the saliency detection model based on the candidate box area and the real labeled area;

[0149] An iterative updating module 305 is used to iteratively update the saliency detection model based on the prediction loss until the model is fitted to obtain an updated saliency detection model;

[0150] The video query module 306 is used to obtain the text to be queried, input the text to be queried into the updated saliency detection model, and obtain the video query result.

[0151] Furthermore, the saliency detection module 302 is specifically used for:

[0152] Video features are extracted from several video frame groups using a pre-trained video feature extraction model to obtain several video frame group feature vectors; text segmentation is performed on the video description text, and text features are extracted from the text segmentation using a pre-trained text feature extraction model to obtain text segmentation feature vectors; feature vector conversion is performed on the video candidate frame to obtain a video candidate frame feature vector; several video frame group feature vectors, text segmentation feature vectors and video candidate frame feature vectors are input into a pre-trained saliency detection model to obtain a salient video frame group.

[0153] Furthermore, the saliency detection module 302 is further used to:

[0154] For each video frame group feature vector, a saliency matrix of each video frame group feature vector is constructed by combining the text segmentation feature vector and the video candidate frame feature vector to obtain several saliency matrices; based on the saliency matrix, a saliency value of each video frame group is calculated; the saliency value is compared with a preset saliency threshold, and a video frame group whose saliency value is greater than the saliency threshold is determined as a salient video frame group.

[0155] Furthermore, the saliency detection module 302 is also used for:

[0156] The feature vector sum of the text word segmentation feature vector and the video candidate frame feature vector is calculated; the inner product of the feature vector of each video frame group and the feature vector sum is calculated to obtain several saliency matrices.

[0157] Furthermore, the predicted loss is an intersection-over-union loss, and the loss prediction module 304 is specifically used for:

[0158] Determine the intersection area of ​​the candidate box area and the true annotation area; and determine the union area of ​​the candidate box area and the true annotation area; calculate the ratio of the intersection area and the union area to obtain the first area intersection-union ratio; calculate the difference between the intersection area and the union area to obtain the difference area; calculate the ratio of the difference area to the union area to obtain the second area intersection-union ratio; based on the first area intersection-union ratio and the second area intersection-union ratio, obtain the prediction loss of the saliency detection model.

[0159] Furthermore, the video processing device 300 further includes a video frame group weighting module, which is specifically used for:

[0160] For each significant video frame group, the significance value of each video frame in the significant video frame group is calculated separately; based on the significance value of each video frame, the average significance value of each significant video frame group is calculated; according to the average significance value of each significant video frame group, a corresponding weight item is configured for each significant video frame group; based on the weight item of each significant video frame group, each significant video frame group is weighted to obtain a weighted significant video frame group.

[0161] Furthermore, the iterative update module 305 is specifically used for:

[0162] The weighted salient video frame group is imported into the saliency detection model for iterative training; in the model iterative training, the prediction loss is back-propagated in the saliency detection model; based on the characteristics of the weighted salient video frame group, the network parameters of the saliency detection model are continuously adjusted until the prediction loss of the saliency detection model is less than the preset loss threshold, thereby obtaining an updated saliency detection model.

[0163] In the above embodiment, the present application discloses a video processing device, which belongs to the field of artificial intelligence technology. The present application obtains video frame groups through frame processing, and introduces a pre-trained saliency detection model, combined with video description text and video candidate frames, to effectively identify and extract salient video frame groups. Then, by comparing the candidate frame area of ​​the salient video frame group with the real annotated area of ​​the video, the prediction loss is accurately calculated, and the saliency detection model is iteratively updated accordingly until the model is highly fitted, which significantly improves the model's positioning accuracy for video clips. The present application solves the conflict problem in similar picture retrieval, achieves clear and accurate video content positioning, optimizes the user experience, and enables users to obtain more accurate and efficient query results when querying specific video content.

[0164] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0165] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0166] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0167] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the video processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0168] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the video processing method.

[0169] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0170] In the above embodiment, the present application discloses a computer device, which belongs to the field of artificial intelligence technology. The present application obtains video frame groups through frame processing, and introduces a pre-trained saliency detection model, combines video description text and video candidate frames, and effectively identifies and extracts salient video frame groups. Then, by comparing the candidate frame area of ​​the salient video frame group with the real annotated area of ​​the video, the prediction loss is accurately calculated, and the saliency detection model is iteratively updated accordingly until the model is highly fitted, which significantly improves the positioning accuracy of the model for video clips. The present application solves the conflict problem in similar picture retrieval, realizes clear and accurate video content positioning, optimizes the user experience, and enables users to obtain more accurate and efficient query results when querying specific video content.

[0171] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the video processing method as described above.

[0172] In the above embodiment, the present application discloses a computer-readable storage medium, which belongs to the field of artificial intelligence technology. The present application obtains video frame groups through frame processing, and introduces a pre-trained saliency detection model, which effectively identifies and extracts salient video frame groups in combination with video description text and video candidate frames. Then, by comparing the candidate frame area of ​​the salient video frame group with the real annotated area of ​​the video, the prediction loss is accurately calculated, and the saliency detection model is iteratively updated accordingly until the model is highly fitted, which significantly improves the positioning accuracy of the model for video clips. The present application solves the conflict problem in similar picture retrieval, realizes clear and accurate video content positioning, optimizes the user experience, and enables users to obtain more accurate and efficient query results when querying specific video content.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0174] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0175] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A video processing method, characterized in that: include: Obtaining a video to be processed, and performing frame processing on the video to be processed to obtain a plurality of video frame groups; Obtaining video description text and video candidate frames, and inputting a plurality of the video frame groups, the video description text and the video candidate frames into a pre-trained saliency detection model to obtain salient video frame groups; Obtaining a candidate frame area corresponding to the salient video frame group, and obtaining a real marked area corresponding to the video to be processed; Calculating the prediction loss of the saliency detection model based on the candidate box area and the true labeled area; Iteratively updating the saliency detection model based on the prediction loss until the model is fitted, thereby obtaining an updated saliency detection model; The text to be queried is obtained, and the text to be queried is input into the updated saliency detection model to obtain a video query result.

2. The video processing method according to claim 1, characterized in that: The step of obtaining the video description text and the video candidate frames, and inputting the plurality of the video frame groups, the video description text and the video candidate frames into a pre-trained saliency detection model to obtain a salient video frame group specifically includes: Extracting video features from the plurality of video frame groups using a pre-trained video feature extraction model to obtain feature vectors of the plurality of video frame groups; Performing text segmentation on the video description text, and using a pre-trained text feature extraction model to perform text feature extraction on the text segmentation to obtain a text segmentation feature vector; Performing feature vector conversion on the video candidate frame to obtain a video candidate frame feature vector; A plurality of the video frame group feature vectors, the text word segmentation feature vectors and the video candidate frame feature vectors are input into the pre-trained saliency detection model to obtain the salient video frame group.

3. The video processing method according to claim 2, characterized in that: The step of inputting a plurality of the video frame group feature vectors, the text segmentation feature vectors and the video candidate frame feature vectors into the pre-trained saliency detection model to obtain the salient video frame group specifically includes: For each of the video frame group feature vectors, a saliency matrix of each of the video frame group feature vectors is constructed by combining the text segmentation feature vector and the video candidate frame feature vector to obtain a plurality of saliency matrices; Based on the saliency matrix, calculating a saliency value of each of the video frame groups; The significance value is compared with a preset significance threshold, and a video frame group whose significance value is greater than the significance threshold is determined as the significant video frame group.

4. The video processing method according to claim 3, characterized in that: The step of constructing a saliency matrix of each of the video frame group feature vectors by combining the text segmentation feature vector and the video candidate frame feature vector to obtain a plurality of saliency matrices specifically includes: Calculate the feature vector sum of the text word segmentation feature vector and the video candidate frame feature vector; The inner product of each of the feature vectors of the video frame group and the sum of the feature vectors is calculated to obtain a plurality of the saliency matrices.

5. The video processing method according to claim 1, characterized in that: The prediction loss is an intersection-over-union loss, and the step of calculating the prediction loss of the saliency detection model based on the candidate box area and the true labeled area specifically includes: Determine an intersection area between the candidate frame area and the real marked area; and determining a union area of ​​the candidate frame area and the true marked area; Calculating a ratio of the intersection area to the union area to obtain a first area intersection-to-union ratio; Calculate the difference between the intersection area and the union area to obtain a difference area; Calculating a ratio of the difference region to the union region to obtain a second region intersection-union ratio; A prediction loss of the saliency detection model is obtained based on the first region intersection-over-union ratio and the second region intersection-over-union ratio.

6. The video processing method according to claim 3, characterized in that: After the step of obtaining the video description text and the video candidate frames, and inputting the plurality of the video frame groups, the video description text and the video candidate frames into a pre-trained saliency detection model to obtain salient video frame groups, the method further includes: For each of the salient video frame groups, separately calculating a saliency value of each video frame in the salient video frame group; Calculating an average significance value of each of the significant video frame groups based on the significance value of each of the video frames; According to the average significance value of each of the significant video frame groups, a corresponding weight item is configured for each of the significant video frame groups; Based on the weight item of each of the significant video frame groups, each of the significant video frame groups is weighted to obtain a weighted significant video frame group.

7. The video processing method according to claim 6, characterized in that: The step of iteratively updating the saliency detection model based on the prediction loss until the model is fitted to obtain an updated saliency detection model specifically includes: Importing the weighted salient video frame group into the saliency detection model for iterative training; In the iterative model training, the prediction loss is back-propagated in the saliency detection model; Based on the features of the weighted salient video frame group, the network parameters of the saliency detection model are continuously adjusted until the prediction loss of the saliency detection model is less than a preset loss threshold, thereby obtaining the updated saliency detection model.

8. A video processing device, characterized in that: include: A frame processing module is used to obtain a video to be processed, and perform frame processing on the video to be processed to obtain a plurality of video frame groups; A saliency detection module, used to obtain video description text and video candidate frames, and input a plurality of the video frame groups, the video description text and the video candidate frames into a pre-trained saliency detection model to obtain a salient video frame group; A region recognition module, used to obtain a candidate frame region corresponding to the salient video frame group, and obtain a real marked region corresponding to the video to be processed; A loss prediction module, used to calculate the prediction loss of the saliency detection model based on the candidate box area and the real labeled area; An iterative updating module, used for iteratively updating the saliency detection model based on the prediction loss until the model is fitted to obtain an updated saliency detection model; The video query module is used to obtain the text to be queried, input the text to be queried into the updated saliency detection model, and obtain the video query result.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the video processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

Images