A method for exploring the contextual relationship of video content based on multi-model fusion algorithm

Abstract

This invention discloses a method for exploring the contextual relationship of video content based on a multi-model fusion algorithm, comprising the following steps: S1, inputting an image and performing image pre-processing; S2, downsampling the result obtained in S1 at least twice; S3, training the result obtained in S2 on three branch models, calculating their respective loss values; S4, summing the loss values ​​obtained in S3 according to corresponding weight ratios; and S5, selecting a branch model as the focus for final inference. This invention utilizes a combination of multiple algorithms to understand video content, thereby deriving the relationship between the preceding and following linkage states of the video, thereby better understanding the video content and solving practical application problems.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for exploring the contextual relationship of video content based on a multi-model fusion algorithm. Background Art

[0002] When scoring video content, it is necessary to accurately locate the targets within the video, the relationship between targets, the relationship within targets, and the linkage relationship between the previous and next frames of the video. Only with these conditions can the entire content of the video be well understood.

[0003] In this scenario, the conventional approach is to understand video content through object detection, which has certain limitations. For example, it cannot understand the relationship between individual objects, the relationship between previous and subsequent video frames, or the previous and subsequent states within a single object. Another approach currently exists, using posture estimation to understand video content. However, posture estimation can only resolve the motion relationships within an object, not the relationships between objects, and thus has certain drawbacks. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for exploring the relationship between the front and back of video content based on a multi-model fusion algorithm, which uses a combination of multiple algorithms to obtain the relationship between the front and back linkage states of the video, thereby better understanding the video content.

[0005] To solve the above technical problems, the present invention provides a method for exploring the contextual relationship of video content based on a multi-model fusion algorithm, comprising the following steps:

[0006] S1. Input the image and perform image pre-processing;

[0007] S2, downsample the result obtained in S1 at least twice;

[0008] S3: Train the three branch models based on the results obtained in S2 and calculate their loss values ​​respectively;

[0009] S4, summing the loss values ​​obtained in S3 according to the corresponding weight ratios;

[0010] S5. Select a branch model as the focus for final reasoning.

[0011] Furthermore, in step S2, before downsampling the image by a factor of two, convolution, pooling, and activation operations are performed on the image.

[0012] Furthermore, in step S3, the three branch models include a detection branch model, a key point branch model and a segmentation branch model. The detection branch model is used for target detection, the key point branch model is used for key point detection and the logical relationship between key points, and the segmentation branch model is used for semantic segmentation and instance segmentation.

[0013] Furthermore, in step S3, the specific steps for calculating the loss value of each branch model are as follows:

[0014] S31, respectively perform calculations on the image using the detection supervision algorithm, the key point supervision algorithm, and the segmentation supervision algorithm, and obtain their loss values ​​respectively;

[0015] S32. Sum the loss values ​​obtained in S31 according to the corresponding weight ratios.

[0016] Furthermore, the detection supervision algorithm is responsible for the positioning supervision of the target in the image, and the detection supervision algorithm includes a classification loss function, a positioning loss function and a confidence loss function.

[0017] Furthermore, the key point supervision algorithm is used to supervise the overall key points and relationship key points, and the key point supervision algorithm includes a binary classification loss function and a multi-classification loss function.

[0018] Furthermore, the formula of the binary classification loss function is:

[0019] loss(x,class)=weight[class](-x[class]+log(∑ j exp(x[j])).

[0020] Furthermore, the formula of the multi-classification loss function is:

[0021]

[0022] Furthermore, the segmentation supervision algorithm is responsible for supervising segmentation to solve the subdivision between objects and the precise coordinate positioning between objects. The loss function formula of the segmentation supervision algorithm is:

[0023] loss(x,class)=weight[class](-x[class]+log(∑ j exp(x[j])).

[0024] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps in the above method are implemented.

[0025] Compared with the prior art, the present invention has at least the following beneficial effects:

[0026] The present invention utilizes a combination of multiple algorithms to understand the video content, thereby being able to obtain the relationship between the previous and subsequent linkage states of the video, thereby better understanding the video content and solving applications in actual scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the process of the method for exploring the contextual relationship of video content based on the multi-model fusion algorithm of the present invention;

[0028] Figure 2 This is a specific flow chart of an embodiment of the method for exploring the contextual relationship of video content based on a multi-model fusion algorithm of the present invention. DETAILED DESCRIPTION

[0029] The following is a more detailed description of the method for exploring the contextual relationship of video content based on a multi-model fusion algorithm, with reference to schematic diagrams. These diagrams illustrate preferred embodiments of the present invention. It should be understood that those skilled in the art may modify the invention described herein while still achieving the beneficial effects of the invention. Therefore, the following description should be understood as generally known to those skilled in the art and not as a limitation of the present invention.

[0030] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are greatly simplified and not to exact scale, and are provided solely for the purpose of assisting in the description of the embodiments of the present invention.

[0031] like Figure 1 As shown, the embodiment of the present invention proposes a method for exploring the contextual relationship of video content based on a multi-model fusion algorithm, including the following steps:

[0032] S1. Input the image and perform image pre-processing;

[0033] S2, downsample the result obtained in S1 at least twice;

[0034] S3: Train the three branch models based on the results obtained in S2 and calculate their loss values ​​respectively;

[0035] S4, summing the loss values ​​obtained in S3 according to the corresponding weight ratios;

[0036] S5. Select a branch model as the focus for final reasoning.

[0037] The following lists preferred embodiments of the method for exploring the context of video content based on a multi-model fusion algorithm to clearly illustrate the content of the present invention. It should be clear that the content of the present invention is not limited to the following embodiments, and other improvements through conventional technical means of ordinary technicians in this field are also within the scope of the concept of the present invention.

[0038] Combined with reference Figure 2 This embodiment proposes a specific operation process of a method for exploring the contextual relationship of video content based on a multi-model fusion algorithm, which is as follows:

[0039] Step 1: Input the image and perform pre-processing on it. The main purpose of pre-processing is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information and simplify the data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition;

[0040] Step 2: After the pre-processing is completed, the obtained results are convolved, pooled, activated, and downsampled to a size that is scaled to 2 times the size;

[0041] Step 3: The result obtained in step 2 is convolved, pooled, and activated again, and downsampled to a size that is 2 times smaller than the original image.

[0042] Step 4: Train three branch models based on the results obtained in step 3. The three branch models are detection branch model, key point branch model and segmentation branch model. Each branch model contains three supervision branches, namely detection supervision algorithm, key point supervision algorithm and segmentation supervision algorithm.

[0043] The detection branch model is used for target detection, the key point branch model is used for key point detection and the logical relationship between key points, and the segmentation branch model is used for semantic segmentation and instance segmentation;

[0044] The detection supervision algorithm is responsible for the positioning supervision of the target in the image. The detection supervision algorithm includes classification loss function, positioning loss function and confidence loss function. There are many corresponding loss functions, which can be selected according to needs;

[0045] The key point supervision algorithm is used to supervise the overall key points and relationship key points. The key point supervision algorithm includes a binary classification loss function and a multi-classification loss function. The formula of the binary classification loss function is:

[0046] loss(x,class)=weight[class](-x[class]+log(∑ j exp(x[j]));

[0047] The formula for the multi-classification loss function is:

[0048]

[0049] The segmentation supervision algorithm is responsible for supervising segmentation to solve the subdivision between objects and the precise coordinate positioning between objects. The loss function formula of the segmentation supervision algorithm is:

[0050] loss(x,class)=weight[class](-x[class]+log(∑ j exp(x[j]));

[0051] Step 5: After the three models are trained, each model obtains three loss values. The weights are set according to different tasks, and the loss values ​​of each model are summed.

[0052] Step 6: Sum the weighted loss values ​​of each model obtained in step 5.

[0053] Step 7: Delete the branch model with poor performance, retain the branch model with the best performance, and perform final reasoning. The selection of the retained branch model is mainly determined by the focus of the problem to be solved, and the weight setting in steps 5 and 6 is also the same. For example, if the main purpose is to ensure the accuracy of target detection, the detection branch model is retained in the final reasoning. The following takes the case of retaining the detection branch model in the final reasoning as an example to explain the weight setting in steps 5 and 6:

[0054] exist Figure 2In the figure, the detect backbone, keypoint backbone, and seg backbone represent the detection branch model, the keypoint branch model, and the segmentation branch model, respectively. The det loss, point loss, and seg loss represent the loss values ​​obtained after the image is respectively calculated by the detection supervision algorithm, the keypoint supervision algorithm, and the segmentation supervision algorithm in the branch model. In the detect backbone, since the final reasoning chooses to retain the detect backbone, the weight ratio of its det loss, point loss, and seg loss is 1:1:1. The sum of the loss values ​​is recorded as sum loss1. Similarly, in the keypoint backbone, since its main task is at the key point and the branch is not used in the final reasoning, the weight ratio is 0.25:1:0.25. The sum of the loss values ​​is recorded as sum loss2. Similarly, in the seg backbone, since its main task is segmentation and the branch is not used in the final reasoning, the weight ratio is 0.25:0.25:1. The sum of the loss values ​​is recorded as sum loss3.

[0055] After the loss value of each branch model is obtained, it is applied to the inference branch through different weight configuration methods (the same setting can be made for each backbone branch during training) and the sum is output. Since the detect backbone is used for final inference, the detect backbone has a larger weight. The weight ratio of sum loss1, sum loss2, and sum loss2 is 0.8:0.1:0.1. After summing, the sum total loss is obtained.

[0056] It should be noted that since only the detect backbone is retained in the final reasoning, the weight ratios of the detect backbone's det loss, point loss, and seg loss are the same during the final reasoning.

[0057] In summary, the present invention has the following advantages over the prior art:

[0058] The present invention utilizes a combination of multiple algorithms to understand the video content, thereby being able to obtain the relationship between the previous and subsequent linkage states of the video, thereby better understanding the video content and solving applications in actual scenarios.

[0059] Accordingly, other embodiments of the present application may further provide a computer-readable storage medium having computer executable instructions stored therein, which implement the various method embodiments of the present application when the computer executable instructions are executed by a processor. Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0060] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for exploring the contextual relationship of video content based on a multi-model fusion algorithm, characterized in that: The steps include: S1. Input the image and perform image pre-processing; S2, downsample the result obtained in S1 at least twice; S3: Train the three branch models based on the results obtained in S2 and calculate their loss values ​​respectively; S4, summing the loss values ​​obtained in S3 according to the corresponding weight ratios; S5. Select a branch model as the focus for final reasoning; Among them, in step S3, the three branch models include a detection branch model, a key point branch model and a segmentation branch model. The detection branch model is used for target detection, the key point branch model is used for key point detection and the logical relationship between key points, and the segmentation branch model is used for semantic segmentation and instance segmentation.

2. The method for exploring the contextual relationship of video content based on a multi-model fusion algorithm according to claim 1, characterized in that: In step S2, before downsampling the image, convolution, pooling, and activation operations are performed on the image.

3. The method for exploring the contextual relationship of video content based on a multi-model fusion algorithm according to claim 1, characterized in that: In step S3, the specific steps for calculating the loss value of each branch model are as follows: S31, respectively perform calculations on the image using the detection supervision algorithm, the key point supervision algorithm, and the segmentation supervision algorithm, and obtain their loss values ​​respectively; S32. Sum the loss values ​​obtained in S31 according to the corresponding weight ratios.

4. The method for exploring the contextual relationship of video content based on a multi-model fusion algorithm according to claim 3, characterized in that: The detection supervision algorithm is responsible for the positioning supervision of the target in the image, and the detection supervision algorithm includes a classification loss function, a positioning loss function and a confidence loss function.

5. The method for exploring the contextual relationship of video content based on a multi-model fusion algorithm according to claim 3, characterized in that: The key point supervision algorithm is used to supervise the overall key points and relationship key points, and the key point supervision algorithm includes a binary classification loss function and a multi-classification loss function.

6. The method for exploring the contextual relationship of video content based on a multi-model fusion algorithm according to claim 5, characterized in that: The formula of the two-class loss function is: loss(x,class)=weight[class](-x[class]+log(∑ j exp(x[j]))。 7. The method for exploring the contextual relationship of video content based on a multi-model fusion algorithm according to claim 5, characterized in that: The formula of the multi-classification loss function is:

8. The method for exploring the contextual relationship of video content based on a multi-model fusion algorithm according to claim 3, wherein: The segmentation supervision algorithm is responsible for supervising segmentation to solve the subdivision between objects and the precise coordinate positioning between objects. The loss function formula of the segmentation supervision algorithm is: loss(x,class)=weight[class](-x[class]+log(∑ j exp(x[j]))。 9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images