Video determination method, apparatus, device, storage medium, and program product

Abstract

This disclosure provides a video determination method, apparatus, device, storage medium, and program product, relating to the field of artificial intelligence technology. The video determination method includes: inputting information describing a target object into a large language model and outputting information about the target object; wherein the information about the target object includes at least two of the following: the motion state of the target object, the motion state of the image acquisition device used to capture the target object, and the scene used to display the target object; inputting the target object into multiple text-based video models respectively, outputting multiple candidate videos, using a multimodal large model to detect the multiple candidate videos, obtaining detection results, wherein the detection results are used to characterize the degree of influence of the content displayed in the multiple candidate videos on the attention of the target object; and determining the target video from the multiple candidate videos based on the detection results.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, this disclosure provides a video determination method, a video determination device, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] Today, thanks to significant advancements in video generation technology, it has become possible to generate product display videos for online transactions.

[0003] In the process of realizing the inventive concept of this disclosure, it was discovered that different application scenarios have different requirements for video display content and display methods. Therefore, the videos generated in the relevant examples are difficult to meet the display requirements of the online transaction field. Summary of the Invention

[0004] In view of this, the present disclosure provides a video determination method, apparatus, device, storage medium, and program product.

[0005] One aspect of this disclosure provides a video identification method, comprising: acquiring a plurality of candidate videos for a target object; wherein the plurality of candidate videos are generated based on information of the target object using a plurality of text-based video models; detecting the plurality of candidate videos using a multimodal large model to obtain detection results, wherein the detection results are used to characterize the degree of influence of the content displayed in the plurality of candidate videos on the attention of the target object; and identifying the target video from the plurality of candidate videos based on the detection results.

[0006] Another aspect of this disclosure provides a video determination apparatus, comprising: acquiring a plurality of candidate videos for a target object; wherein the plurality of candidate videos are generated based on information of the target object using a plurality of text-based video models; detecting the plurality of candidate videos using a multimodal large model to obtain detection results, wherein the detection results are used to characterize the degree of influence of the content displayed in the plurality of candidate videos on the attention of the target object; and determining the target video from the plurality of candidate videos based on the detection results.

[0007] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the methods described above.

[0008] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.

[0009] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0010] According to embodiments of this disclosure, by employing a multimodal large model to detect multiple candidate videos generated by the textual video model, display videos suitable for online transaction scenarios are selected. Therefore, this at least partially overcomes the problem that videos generated in related examples are difficult to meet the display requirements of the online transaction field, thereby improving the efficiency of detecting videos generated by the textual video model and further enhancing the matching degree between the content displayed in the target video and the display requirements of the target object in the online transaction scenario. Attached Figure Description

[0011] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of this disclosure with reference to the accompanying drawings, in which:

[0012] Figure 1 This illustration schematically shows an exemplary system architecture to which the video determination method and apparatus of this disclosure can be applied;

[0013] Figure 2 A flowchart illustrating a video determination method according to an embodiment of the present disclosure is shown schematically;

[0014] Figure 3 The illustration shows a schematic diagram of the principle of a video determination method according to an embodiment of the present disclosure;

[0015] Figure 4 The illustration shows a schematic diagram of the principle of a video determination method according to another embodiment of the present disclosure.

[0016] Figure 5 The schematic diagram illustrates the principle of product category detection according to another embodiment of the present disclosure;

[0017] Figure 6 The schematic diagram illustrates the principle of integrity detection according to another embodiment of the present disclosure;

[0018] Figure 7 The schematic diagram illustrates the principle of inter-frame difference detection according to another embodiment of the present disclosure;

[0019] Figure 8 A block diagram of a video determination apparatus according to an embodiment of the present disclosure is schematically shown; and

[0020] Figure 9 A block diagram of an electronic device suitable for implementing a video determination method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0021] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] Video plays an increasingly important role in online shopping transactions, especially with the rise of various shopping application platforms. Video is gradually becoming an indispensable part of online transactions. Video has the following functions in online transaction scenarios:

[0026] 1. Enhance product presentation: Videos can vividly showcase the appearance, functions, and usage of products, allowing consumers to understand them more intuitively.

[0027] 2. Improve conversion rates: Research shows that product pages containing videos generally have higher conversion rates than pages with only images. Videos effectively attract customer attention and increase their desire to buy.

[0028] 3. Enhance user experience: Through videos, consumers can obtain more comprehensive information, reduce doubts about products, and thus enhance the shopping experience.

[0029] On the other hand, video generation technology has made significant progress. Generating product demonstration videos for online transactions has become possible. How to design an automated method to evaluate candidate videos has become a pressing problem in the application of video generation in online transactions.

[0030] However, existing video evaluation technologies cannot be directly applied to the evaluation of online transaction videos because these videos have their own unique characteristics and require further testing. For example, online transaction videos showcase products and their selling points. A single video needs to display not only parts of the product but also an overall image of it. Furthermore, products often contain a lot of fine-grained category information. While most focus on broad categories like animals and people, this disclosure aims to address fine-grained, multi-layered target audiences, such as wireless in-ear headphones, rather than simply any headphones.

[0031] In the commercial film industry and video portal websites, or in scenarios where small-batch video generation is being evaluated, this may be done manually by professionals, which is often expensive and not suitable for large-scale applications that require low-cost online transaction video generation and evaluation.

[0032] In view of the above, embodiments of this disclosure provide a video determination method, including: acquiring multiple candidate videos for a target object; wherein the multiple candidate videos are generated based on information of the target object using multiple text-based video models; detecting the multiple candidate videos using a multimodal large model to obtain detection results, wherein the detection results are used to characterize the degree of influence of the content displayed in the multiple candidate videos on the attention of the target object; and determining the target video from the multiple candidate videos based on the detection results.

[0033] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0034] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0035] Figure 1 An exemplary system architecture 100 for which video determination methods and apparatus can be applied according to embodiments of this disclosure is illustrated schematically. It should be noted that... Figure 1The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0038] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the video determination method provided in this embodiment can generally be executed by server 105. Correspondingly, the video determination device provided in this embodiment can generally be located in server 105. The video determination method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the video determination device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Alternatively, the video determination method provided in this embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Correspondingly, the video determination device provided in this embodiment can also be located in terminal devices 101, 102, or 103, or in other terminal devices different from terminal devices 101, 102, or 103.

[0041] For example, the candidate video may originally be stored in any one of terminal devices 101, 102, or 103 (e.g., terminal device 101, but not limited thereto), or it may be stored on an external storage device and imported into terminal device 101. Then, terminal device 101 may execute the video determination method provided in the embodiments of this disclosure locally, or send the video to be processed to other terminal devices, servers, or server clusters, and have the other terminal devices, servers, or server clusters that receive the image to be processed execute the video determination method provided in the embodiments of this disclosure.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] Figure 2 A flowchart illustrating a video determination method according to an embodiment of the present disclosure is shown schematically.

[0044] like Figure 2 As shown, the method includes operations S210~S230.

[0045] In operation S210, multiple candidate videos for the target object are acquired.

[0046] When operating the S220, a multimodal large model is used to detect multiple candidate videos and obtain detection results.

[0047] In operation S230, the target video is determined from multiple candidate videos based on the detection results.

[0048] According to embodiments of this disclosure, the target object can be an item to be displayed on an online trading website, and multiple candidate videos can be generated using a text-based video model based on information used to describe the target object.

[0049] In some embodiments, the information used to describe the target object may include, but is not limited to: the name of the item, its color, the scene to be displayed, the shape of the item, and other information.

[0050] In some embodiments, the text-based video model can be any open-source model capable of generating video from text. This disclosure does not limit the text-based video model.

[0051] In some embodiments, different open-source text-based video models can be used to independently generate multiple candidate videos based on the same text prompt. These text-based video models are independent of each other and do not affect each other.

[0052] According to embodiments of this disclosure, a multimodal large model can be used to perform multi-dimensional detection on multiple candidate videos. The detection dimensions may include, but are not limited to: whether the items displayed in the candidate video correspond to the target object; whether the candidate video clearly displays the target object; the frame rate and angle of the target object displayed in the candidate video, and whether the details or usage scenario of the target object are clearly displayed; whether the video is continuous and free from distortion when displaying the target object, etc. This disclosure does not limit the detection dimensions.

[0053] In some embodiments, corresponding text instructions can be configured based on detection dimensions. These text instructions may include at least the detection dimension and a corresponding score. Then, the text instructions and each candidate video are input into a multimodal large model, outputting detection results corresponding to each detection dimension. These detection results can be the scores corresponding to that detection dimension.

[0054] Then, for the above multi-dimensional detection, the multiple detection results can be calculated using the weighted average method, evidence fusion method, fuzzy comprehensive evaluation method, decision tree method, or normalization method to obtain the comprehensive detection result for each candidate video.

[0055] In operating S230, target videos suitable for online transaction scenarios can be selected based on the comprehensive detection results of each candidate video.

[0056] For example, a detection result threshold can be set, and candidate videos that exceed the detection result threshold can be used as target videos; or a ranking method can be used, and the top X candidate videos can be used as target videos.

[0057] According to embodiments of this disclosure, by employing a multimodal large model to detect multiple candidate videos generated by the textual video model, display videos suitable for online transaction scenarios are selected. Therefore, this at least partially overcomes the problem that videos generated in related examples are difficult to meet the display requirements of the online transaction field, thereby improving the efficiency of detecting videos generated by the textual video model.

[0058] According to embodiments of this disclosure, the detection results may include: a first detection result, a second detection result, and a third detection result. Detecting multiple candidate videos using a multimodal large model to obtain detection results may include the following operations: performing product category detection on the multiple candidate videos using the multimodal large model to obtain a first detection result; performing completeness detection on the multiple candidate videos using the multimodal large model to obtain a second detection result; and performing inter-frame difference detection on the multiple candidate videos using the multimodal large model to obtain a third detection result.

[0059] In this embodiment of the disclosure, the first detection result characterizes the degree of matching between the content and the category to which the target object belongs. The second detection result characterizes whether the content includes sub-content used to fully display the target object. The third detection result characterizes the degree of difference between the content displayed in multiple consecutive video frames in each candidate video.

[0060] Figure 3 The illustration shows a schematic diagram of the principle of a video determination method according to an embodiment of the present disclosure.

[0061] The following is for reference. Figure 3 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0062] First, Prompt310 is used as input, and multiple video generation models are used to generate multiple candidate videos. Specifically, the same Prompt310 is input into textual video model M1320, textual video model M2321, ..., textual video model M... n Among the 322 n text-based video models, these models will output multiple candidate videos, including: video V1330, video V2331, ..., video V n There are n videos in total (332).

[0063] Then, the multimodal large model 340 is used to perform product category detection on the multiple candidate videos obtained in operation S210 to obtain a category consistency score 350; where the category consistency score 350 represents the degree of matching between the video content and the category to which the target object belongs;

[0064] A multimodal large model 341 was used to perform completeness detection on multiple candidate videos, resulting in a completeness score 351. The completeness score 351 characterizes whether the video content includes sub-content used to fully display the target object.

[0065] The inter-frame difference detection of multiple candidate videos is performed using a multimodal large model 342 to obtain an inter-frame consistency score 352; where the inter-frame consistency score 352 represents the degree of difference between the content displayed in multiple consecutive video frames in each candidate video.

[0066] Finally, the category consistency score (350), integrity score (351), and inter-frame consistency score (352) of each video are used as the detection result (360) for each video.

[0067] According to embodiments of this disclosure, the category consistency score 350, integrity score 351, inter-frame consistency score 352, and detection result 360 are different for each candidate video.

[0068] In operation S230, candidate videos are detected based on the detection results obtained in operation S220, and target videos suitable for online transaction scenarios are selected.

[0069] According to embodiments of this disclosure, candidate videos are evaluated from three dimensions: category matching degree, completeness, and inter-frame consistency. This allows for the selection of display videos suitable for online transaction scenarios, further improving the matching degree between video display content and the display needs of online transaction scenarios for target audiences.

[0070] In some embodiments, to further improve the matching degree between the display content of the video output by the Wensheng video model and the display requirements of online transaction scenarios, information such as the motion state of the target object, the scene atmosphere, and the camera's motion state relative to the target object can be added to the prompt text input by the Wensheng video model. This information can be pre-configured or can be generated flexibly using the natural language understanding capabilities of a large language model to flexibly generate corresponding display actions, display scenes, and other prompt text for the display requirements of different items.

[0071] Therefore, the above method may also include the following operations: inputting information used to describe the target object into a large language model and outputting information about the target object; wherein, the information about the target object includes at least two of the following: the motion state of the target object, the motion state of the image acquisition device used to capture the target object, and the scene used to display the target object; and inputting the target object into multiple text-based video models respectively and outputting multiple candidate videos.

[0072] Figure 4 The illustration shows a schematic diagram of the principle of a video determination method according to another embodiment of the present disclosure.

[0073] The following is for reference. Figure 4 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0074] According to embodiments of this disclosure, any open-source large language model can be used for the large language model, and this disclosure does not limit the large language model used.

[0075] like Figure 4 As shown, first, obtain all first-level categories of products for sale in online transactions covering the entire network, and then select several products from each first-level category as the subject.

[0076] For example, eyeglasses can be categorized into sunglasses, swimming goggles, and safety goggles, while slippers can be categorized into flip-flops and cotton slippers. These subcategories are used to evaluate the video generation model's ability to recognize and differentiate similar product categories. To better showcase the complete form and details of the target object, the object's motion state and changes in camera motion can be input into the large language model 420 as information 410 describing the target object. This information 410 can include, but is not limited to, 360-degree rotation, zooming from a distance, and panning from left to right. Furthermore, cues can be enhanced with specific scene and video-specific atmospheric effects to ensure the generated video better suits the e-commerce context. Finally, the large language model 420 outputs a comprehensive and detailed Prompt 310.

[0077] According to embodiments of this disclosure, a large language model is used to output scene information such as atmosphere and lighting suitable for product display, as well as product actions and camera actions. Based on the large language model's full understanding of product information, it can flexibly adapt corresponding display scenes and actions for different products, so that the Wensheng video model can better understand the display needs of the target object in the online transaction scenario, and further improve the quality of video generation.

[0078] Figure 5 The illustration shows a schematic diagram of the principle of product category detection according to another embodiment of the present disclosure.

[0079] According to embodiments of this disclosure, when using a multimodal large model to perform product category detection on multiple candidate videos to obtain a first detection result, the following operations may be included: inputting multiple candidate videos and multi-level category evaluation reference information as first prompts into the multimodal large model, and outputting the first detection result.

[0080] In the embodiments of this disclosure, the multi-level category evaluation reference information is determined based on the hierarchical relationship of item category attributes.

[0081] For example, the multi-level category evaluation reference information can be divided into 5 levels. The higher the score, the more consistent the corresponding category strength. These 5 levels can be: 0 points: Level 1 category inconsistent; 1 point: Level 1 category consistent, Level 2 category inconsistent; 2 points: Level 1 category consistent, Level 2 category consistent, Level 3 category inconsistent; 3 points: Level 1 category consistent, Level 2 category consistent, Level 3 category consistent, Level 4 category inconsistent; 4 points: All categories consistent.

[0082] like Figure 5 As shown, product category detection is a multi-level category consistency test between products appearing in candidate videos and target objects. Specifically: First, multi-level category evaluation reference information 510 is formulated, and then it is input into the multimodal large model 530 along with the candidate videos 520. The multimodal large model 530 will output the score based on the multi-level category evaluation reference information 510 and indicate the reason for such scoring 540, and finally obtain the category consistency score 550.

[0083] like Figure 5 In addition to the category consistency score, the information output by the multimodal large model can also include the reason for the score. For example, the category consistency score is "1" because black flip-flops and black slippers both belong to the same level 1 category: slippers, but the level 2 categories are different, so the score is "1".

[0084] In this embodiment, the multi-level category evaluation reference information can be configured based on the needs of the application scenario. This embodiment does not limit the grading method or level. The multimodal large model 530 can be any large model capable of processing multimodal information including text and video. This embodiment does not specifically limit the multimodal large model.

[0085] In the embodiments of this disclosure, product category detection is performed by constructing product categories and using a multimodal large model to detect candidate videos. This ensures that the selected target videos accurately showcase the characteristics of the sub-categories of the target product, making it more suitable for the display needs of online transaction products. Furthermore, by utilizing a multimodal large model based on multi-level category evaluation reference information and candidate videos as prompt text, detection is applicable to large-scale candidate video detection with high efficiency.

[0086] Figure 6 The schematic diagram illustrates the principle of integrity detection according to another embodiment of the present disclosure.

[0087] According to embodiments of this disclosure, the process of using a multimodal large model to perform integrity detection on multiple candidate videos and obtain a second detection result may include the following operations: extracting frames from each candidate video to obtain multiple target frames; and using a multimodal large model to perform integrity detection on the multiple target frames to obtain a second detection result for each candidate video.

[0088] In the embodiments of this disclosure, the second detection result can be detected from the dimension of the completeness of the target object display in the candidate video. Specifically, it can be done by extracting frames from the candidate video and detecting which frames in these extracted target frames completely display the target object. Furthermore, it can also detect the ratio of the number of video frames in the candidate video that display the target object in detail to the number of video frames that completely display the target object.

[0089] In embodiments of this disclosure, the second detection result for each candidate video includes a first detection sub-result and a second detection sub-result.

[0090] The integrity detection of multiple target frames is performed using a multimodal large model to obtain a second detection result for each candidate video. This can include the following operations: inputting multiple target frames and first information as second prompts into the multimodal large model and outputting a first detection sub-result; and inputting multiple target frames and second information as third prompts into the multimodal large model and outputting a second detection sub-result.

[0091] In this embodiment of the disclosure, the first detection sub-result is used to characterize whether the content includes sub-content used to fully display the target object. The second detection sub-result is used to characterize the proportion of the sub-content used to fully display the target object within the content.

[0092] like Figure 6 As shown, firstly, frames are extracted from each candidate video to obtain multiple target frames 620. Then, a multimodal large model is used to perform integrity detection on the target frames 620, obtaining an integrity score 680 for each candidate video.

[0093] Specifically, during the integrity detection, the target frame 620 and Prompt A610 are input into the multimodal large model 640 to obtain the integrity score A660, while Prompt B630 is input into the multimodal large model 650 to obtain the integrity score B670; both integrity scores A660 and B670 belong to the integrity score 680.

[0094] According to embodiments of this disclosure, online transaction videos are required not only to display details, such as improvements that differentiate them from similar or older products, but also to display overall visual information about the product.

[0095] Therefore, the first step is to detect whether the candidate video contains video frames that globally display the target object.

[0096] For example, when extracting frames, the target frame can be set to 10 frames. For each frame, a multimodal large model is input for question-and-answer processing. Prompt A610 can be used to check whether the product is fully presented in the video frame using the large model, and Prompt B630 can be used to check the number of frames in the video frame that show the complete product using the large model. Then, the multimodal large model is used for judging, and the completeness scores A660 and B670 are output respectively. That is to say, if the candidate video does not show the overall image of the target object in all target frames, the completeness score A660 is 0, and otherwise it is 1.

[0097] In addition, candidate videos that showcase the target object must be differentiated in terms of degree, because some details of the target object must also be shown. It is not acceptable for candidate videos to only show the overall picture of the target object. Here, the completeness score B670 is used to differentiate the proportion of the completeness of the candidate videos. The output value of the completeness score B670 will be between 0 and 10.

[0098] For example, different mapping relationships between ratios and output values ​​can be pre-configured. These mapping relationships can be configured according to the needs of actual application scenarios, and this disclosure does not impose specific limitations on them.

[0099] According to embodiments of this disclosure, the completeness detection is used to detect whether the candidate video contains video frames that can fully display the target object, and the ratio of the number of video frames used to display the target object in detail and globally, thereby determining the degree to which the candidate video displays the target object, so as to further improve the content displayed by the final output target video to be more suitable for the display needs of the target object in online transaction scenarios.

[0100] Figure 7 The schematic diagram illustrates the principle of inter-frame difference detection according to another embodiment of the present disclosure.

[0101] According to embodiments of this disclosure, in order to perform inter-frame difference detection on multiple candidate videos using a multimodal large model to obtain a third detection result, the method may include: extracting multiple keyframes of each candidate video; inputting the multiple keyframes and the attributes of the target object as a fourth cue into the multimodal large model, and outputting the region information in each keyframe used to display the target object; and processing the region information to obtain a third detection result for each candidate video.

[0102] In some embodiments, the attributes of the target object may include basic attributes such as the item's color and category. For example: a blue umbrella.

[0103] In some embodiments, a multimodal large model can be used to segment each keyframe into regions, thereby obtaining information about the regions in the keyframe used to display the target object, such as the coordinates of the masked region. Then, the inter-frame consistency of the candidate videos can be determined by comparing the information about the masked regions in the keyframes of each candidate video.

[0104] For example, processing region information to obtain a third detection result for each candidate video can include the following operations: determining a first target keyframe from multiple keyframes based on region information; wherein the area of ​​the region used to display the target object in the first target keyframe is greater than a first threshold; dividing each keyframe into grids to obtain region information for displaying the target object within each grid of each keyframe; determining a second target keyframe from multiple keyframes based on the region information for displaying the target object within each grid; wherein the difference between the region used to display the target object within each grid of the second target keyframe and the region used to display the target object within each grid of adjacent frames is greater than a second threshold; and obtaining a third detection result for each candidate video frame based on the number of first target keyframes, the number of second target keyframes, and the total number of each candidate video frame.

[0105] According to embodiments of this disclosure, such as Figure 7 As shown, firstly, multiple keyframes 710 are extracted from each candidate video. Then, the keyframes 710 and the attributes 720 of the target object are input into the multimodal large model 730, and the region information 740 used to display the target object in each keyframe is output. Finally, the region information is processed to obtain the inter-frame consistency score 790 for each candidate video.

[0106] According to embodiments of this disclosure, firstly, a set of keyframes 710 from the candidate video and the attributes 720 of the target object are input into a pre-trained multimodal large model to segment the product and obtain a predicted mask. The multimodal large model can be any model capable of recognizing object regions; this disclosure does not specifically limit its implementation. The predicted mask is used to segment the target object from the corresponding original keyframes. The attributes 720 of the target object include key information such as the type and color of the target object's product.

[0107] Next, the effective mask area for each frame is calculated and compared to a threshold T1 set from empirical observations. The number of frames exceeding the threshold T1 is divided by N to calculate the Score eff (effective score). Here, the Effectiveness Calculator 750 is used to perform the calculation of the effective score and outputs EFF0760, where EFF0 represents the effective score corresponding to the 0th keyframe. The effective scores for all keyframes are calculated in this manner, for example, EFF1761. Here, threshold T1 represents the effective mask area threshold, and N represents the total number of frames in the corresponding candidate video.

[0108] Next, each keyframe 710 is divided into a grid using grid size G and threshold T2, and the mask area difference of each grid between consecutive frames is measured. Grids with differences exceeding threshold T2 are counted, and the results are compared with another threshold T3 to determine the number of frames with excessive variation. The total number of differences exceeding threshold T3 is divided by N−1 to obtain the score diff, for example: Diffi 770.

[0109] Finally, the valid scores can be taken. and difference scores The average value is 780, resulting in an inter-frame consistency score of 790. Threshold T2 represents the mask area difference threshold for each grid. Threshold T3 represents the number of grids whose mask area difference exceeds threshold T2.

[0110] According to embodiments of this disclosure, the aforementioned inter-frame difference detection first assesses the accuracy of the product theme in the generated candidate video. If the theme does not match the attribute 720 of the input target object, the multimodal large model 730 will be unable to accurately segment the product. Secondly, the inter-frame difference detection also assesses whether there are any abrupt changes between video frames. Finally, the inter-frame consistency score 790 is obtained by combining the above two outputs. The resulting target video can more vividly and realistically display the appearance of the target object, thereby enhancing the attractiveness of the target object.

[0111] Figure 8 A block diagram of a video determination apparatus according to an embodiment of the present disclosure is shown schematically.

[0112] like Figure 8 As shown, the video determination device 800 includes an acquisition module 810, a detection module 820, and a determination module 830.

[0113] The acquisition module 810 is used to acquire multiple candidate videos for the target object; wherein, the multiple candidate videos are generated based on the information of the target object using multiple text-based video models.

[0114] The detection module 820 is used to detect multiple candidate videos using a multimodal large model and obtain detection results. The detection results are used to characterize the degree of influence of the content displayed in the multiple candidate videos on the attention of the target object.

[0115] The determination module 830 is used to determine the target video from multiple candidate videos based on the detection results.

[0116] According to embodiments of this disclosure, the detection module 820 may include a first detection submodule, a second detection submodule, and a third detection submodule.

[0117] The first detection submodule is used to perform product category detection on multiple candidate videos using a multimodal large model to obtain the first detection result; wherein, the first detection result represents the matching degree between the content and the category to which the target object belongs.

[0118] The first detection submodule includes a first detection unit, which is used to input multiple candidate videos and multi-level category evaluation reference information as first prompts into a multimodal large model and output the first detection result; wherein, the multi-level category evaluation reference information is determined based on the hierarchical relationship of the item category attributes.

[0119] According to embodiments of this disclosure, the second detection submodule is used to perform completeness detection on multiple candidate videos using a multimodal large model to obtain a second detection result; wherein, the second detection result characterizes whether the content includes sub-content used to fully display the target object.

[0120] According to embodiments of this disclosure, the second detection submodule may include a first target frame unit and a second detection unit.

[0121] The first target frame unit is used to extract frames from each candidate video to obtain multiple target frames.

[0122] The second detection unit is used to perform integrity detection on multiple target frames using a multimodal large model, and obtain a second detection result for each candidate video.

[0123] According to embodiments of this disclosure, the second detection unit may include a first detection subunit and a second detection subunit.

[0124] The first detection subunit is used to take multiple target frames and the first information as the second prompt input to the multimodal large model and output the first detection sub-result; wherein, the first detection sub-result is used to characterize whether the content includes sub-content used to fully display the target object.

[0125] The second detection subunit is used to input multiple target frames and second information as the third prompt into a multimodal large model and output the second detection sub-result; wherein, the second detection sub-result is used to characterize the proportion of sub-content used to fully display the target object.

[0126] According to embodiments of this disclosure, a third detection submodule is used to perform inter-frame difference detection on multiple candidate videos using a multimodal large model to obtain a third detection result; wherein, the third detection result characterizes the degree of difference between the content displayed in multiple consecutive video frames in each candidate video.

[0127] According to embodiments of this disclosure, the third detection submodule may include a first keyframe unit, a third detection unit, and a first processing unit.

[0128] The first keyframe unit is used to extract multiple keyframes from each candidate video.

[0129] The third detection unit is used to take the attributes of multiple keyframes and the target object as the fourth cue input to the multimodal large model, and output the region information of each keyframe used to display the target object.

[0130] The first processing unit is used to process the regional information to obtain the third detection result for each candidate video.

[0131] According to embodiments of this disclosure, the first processing unit may include a first keyframe subunit, a first grid division subunit, a first determination subunit, and a third detection subunit.

[0132] The first keyframe subunit is used to determine the first target keyframe from multiple keyframes based on region information; wherein the area of ​​the region used to display the target object in the first target keyframe is greater than a first threshold.

[0133] The first grid division sub-unit is used to divide each keyframe into grids, obtaining the area information of the target object within each grid of each keyframe.

[0134] The first determining sub-unit is used to determine the second target key frame from multiple key frames based on the area information of the target object in each grid; wherein the difference between the area of ​​the target object in each grid of the second target key frame and the area of ​​the target object in each grid of the adjacent frame is greater than a second threshold.

[0135] The third detection subunit is used to obtain a third detection result for each candidate video frame based on the number of first target keyframes, the number of second target keyframes, and the total number of each candidate video frame.

[0136] According to embodiments of this disclosure, the above-described apparatus further includes a first information output module and a first video generation module.

[0137] The first information output module is used to input information describing the target object into the large language model and output the information of the target object; wherein, the information of the target object includes at least two of the following: the motion state of the target object, the motion state of the image acquisition device used to capture the target object, and the scene used to display the target object.

[0138] The first video generation module is used to input the target object into multiple text-based video models and output multiple candidate videos.

[0139] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0140] For example, any plurality of the acquisition module 810, detection module 820, and determination module 830 may be combined into one module / unit / subunit, or any one of these modules / units / subunits may be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits may be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the acquisition module 810, detection module 820, and determination module 830 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 810, the detection module 820, and the determination module 830 may be implemented at least partially as a computer program module that can perform corresponding functions when the computer program module is run.

[0141] It should be noted that the video determination device part in the embodiments of this disclosure corresponds to the video determination method part in the embodiments of this disclosure. For a detailed description of the video determination device part, please refer to the video determination method part, which will not be repeated here.

[0142] Figure 9 A block diagram of an electronic device suitable for implementing a video determination method according to an embodiment of the present disclosure is shown schematically. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0143] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0144] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0145] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The system 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0146] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by processor 901, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0147] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0148] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0149] For example, according to embodiments of this disclosure, a computer-readable storage medium may include one or more memories other than the ROM 902 and / or RAM 903 described above.

[0150] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0151] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0152] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0153] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0155] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A video determination method, comprising: Obtain multiple candidate videos for a target object; wherein, the multiple candidate videos are generated based on information of the target object using multiple text-based video models; A multimodal large model is used to detect the multiple candidate videos, obtaining detection results. These detection results characterize the influence of the content displayed in the multiple candidate videos on the attention level of the target object. The target video is determined from the plurality of candidate videos based on the detection results.

2. The method according to claim 1, wherein, The test results include: a first test result, a second test result, and a third test result; The detection of the multiple candidate videos using a multimodal large model to obtain detection results includes: The multimodal large model is used to perform product category detection on the multiple candidate videos to obtain a first detection result; wherein, the first detection result represents the matching degree between the content and the category to which the target object belongs; The multimodal large model is used to perform completeness detection on the multiple candidate videos to obtain a second detection result; wherein, the second detection result characterizes whether the content includes sub-content used to fully display the target object; and The multimodal large model is used to perform inter-frame difference detection on the multiple candidate videos to obtain a third detection result; wherein, the third detection result represents the degree of difference between the content displayed in multiple consecutive video frames in each candidate video.

3. The method according to claim 2, wherein, The step of using the multimodal large model to perform product category detection on the multiple candidate videos to obtain a first detection result includes: The multiple candidate videos and multi-level category evaluation reference information are used as the first prompts, input into the multimodal large model, and the first detection result is output. The multi-level category evaluation reference information is determined based on the hierarchical relationship of item category attributes.

4. The method according to claim 2, wherein, The step of using the multimodal large model to perform integrity detection on the multiple candidate videos to obtain a second detection result includes: Frames are extracted from each candidate video to obtain multiple target frames; The multimodal large model is used to perform integrity detection on the multiple target frames to obtain a second detection result for each candidate video.

5. The method according to claim 4, wherein, The second detection result for each candidate video includes a first detection sub-result and a second detection sub-result; The step of using the multimodal large model to perform integrity detection on the multiple target frames to obtain a second detection result for each candidate video includes: The multiple target frames and the first information are used as a second prompt input to the multimodal large model, and a first detection sub-result is output; wherein, the first detection sub-result is used to characterize whether the content includes sub-content for fully displaying the target object; and The multiple target frames and the second information are used as the third prompt input to the multimodal large model, and the second detection sub-result is output; wherein, the second detection sub-result is used to characterize the proportion of the sub-content used to fully display the target object within the model.

6. The method according to claim 2, wherein, The step of using the multimodal large model to perform inter-frame difference detection on the multiple candidate videos to obtain a third detection result includes: Extract multiple keyframes from each candidate video; The multimodal large model is input with the multiple keyframes and the attributes of the target object as a fourth cue, and the output is the region information in each keyframe used to display the target object; and The region information is processed to obtain a third detection result for each candidate video.

7. The method according to claim 6, wherein, The process of processing the region information to obtain a third detection result for each candidate video includes: Based on the region information, a first target keyframe is determined from the plurality of keyframes; wherein, the area of ​​the region in the first target keyframe used to display the target object is greater than a first threshold. Each keyframe is divided into grids to obtain the area information of the target object within each grid of each keyframe; Based on the region information within each grid used to display the target object, a second target keyframe is determined from the plurality of keyframes; wherein the difference between the region within each grid of the second target keyframe used to display the target object and the region within each grid of an adjacent frame used to display the target object is greater than a second threshold; and Based on the number of the first target keyframes, the number of the second target keyframes, and the total number of each candidate video frame, a third detection result is obtained for each candidate video frame.

8. The method according to any one of claims 1-7, wherein, The method further includes: Information describing the target object is input into a large language model, and the model outputs information about the target object; wherein, the information about the target object includes at least two of the following: the motion state of the target object, the motion state of the image acquisition device used to capture the target object, and the scene used to display the target object; and The target object is input into multiple text-based video models, and the multiple candidate videos are output.

9. A video determining device, comprising: The acquisition module is used to acquire multiple candidate videos for a target object; wherein, the multiple candidate videos are generated based on the information of the target object using multiple text-based video models; The detection module is used to detect the multiple candidate videos using a multimodal large model to obtain detection results, wherein the detection results are used to characterize the degree of influence of the content displayed in the multiple candidate videos on the attention of the target object; and A determination module is used to determine the target video from the plurality of candidate videos based on the detection results.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images