Model training, prop effect determination method and device, equipment and storage medium

Abstract

This specification provides a model training method, apparatus, device, and storage medium for determining the effectiveness of props. The model training method includes: constructing training samples in the form of triples; the training samples include a baseline video frame, positive example video frames, and negative example video frames; the baseline video frame and the positive example video frames have the same effective props; the baseline video frame and the negative example video frames have different effective props; using the training samples in the form of triples and a preset loss function, a pre-built neural network structure is trained; the trained neural network structure is used as a prop fingerprint model; the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to a target video frame to determine whether the prop in the target video frame is effective. Through this embodiment, it is possible to accurately determine whether the props added to the target video are effective.

Description

Technical Field

[0001] This document relates to the field of video processing technology, and in particular to a method, apparatus, device, and storage medium for model training and determining the effectiveness of props. Background Technology

[0002] Users can create short videos using various applications. When creating short videos, users can add various props as needed, such as headwear, backgrounds, borders, and clothing. However, after a user adds a prop, it may or may not work. For example, if the algorithm cannot detect the user's face, the added headwear or similar props will not work. Currently, the backend relies on the user's prop-adding action to determine whether props in a video are effective. Once the prop-adding action is detected, it is considered effective. As described above, this method of determining prop effectiveness has a low accuracy rate. Summary of the Invention

[0003] This specification provides a model training method, apparatus, device, and storage medium for determining the effectiveness of props added to a video, which can accurately determine whether the props added to the video are effective.

[0004] In a first aspect, embodiments of this specification provide a model training method, comprising: constructing training samples in the form of triplets; the training samples include a baseline video frame, positive example video frames, and negative example video frames; the baseline video frame and the positive example video frames have the same effective props; the baseline video frame and the negative example video frames have different effective props; training a pre-built neural network structure using the training samples in the form of triplets and a preset loss function; the loss function is used to make the distance between the prop fingerprint vector corresponding to the baseline video frame and the prop fingerprint vector corresponding to the positive example video frame less than a first preset distance, and to make the distance between the prop fingerprint vector corresponding to the baseline video frame and the prop fingerprint vector corresponding to the negative example video frame greater than a second preset distance; using the trained neural network structure as a prop fingerprint model; the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to a target video frame to determine whether the prop in the target video frame is effective; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0005] Secondly, embodiments of this specification provide a method for determining the effectiveness of a prop, comprising: acquiring a target video frame for determining whether a prop is effective; processing the target video frame using a prop fingerprint model trained as described in the first aspect to obtain a first prop fingerprint vector corresponding to the target video frame; the first prop fingerprint vector representing the image features of the prop in the target video frame; acquiring a pre-established prop fingerprint vector library; the prop fingerprint vector library storing multiple second prop fingerprint vectors; the second prop fingerprint vectors being obtained by processing the pre-acquired prop-effective video frames using the prop fingerprint model, representing the image features of the prop in the prop-effective video frames; and determining whether the prop in the target video frame is effective based on the first prop fingerprint vector and the second prop fingerprint vector.

[0006] Thirdly, embodiments of this specification provide a model training apparatus, comprising: a sample construction unit for constructing training samples in the form of triplets; the training samples include a baseline video frame, positive example video frames, and negative example video frames; the baseline video frame and the positive example video frames have the same active props; the baseline video frame and the negative example video frames have different active props; a model training unit for training a pre-built neural network structure using the training samples in the form of triplets and a preset loss function; the loss function is used to make the distance between the prop fingerprint vector corresponding to the baseline video frame and the prop fingerprint vector corresponding to the positive example video frame less than a first preset distance, and to make the distance between the prop fingerprint vector corresponding to the baseline video frame and the prop fingerprint vector corresponding to the negative example video frame greater than a second preset distance; and a model generation unit for using the trained neural network structure as a prop fingerprint model; the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to a target video frame to determine whether the prop in the target video frame is active; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0007] Fourthly, embodiments of this specification provide a prop activation determination device, comprising: a first acquisition unit, configured to acquire a target video frame for determining whether a prop is active, and process the target video frame using a prop fingerprint model trained as described in the third aspect to obtain a first prop fingerprint vector corresponding to the target video frame; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame; a second acquisition unit, configured to acquire a pre-established prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vectors are obtained by processing the pre-acquired prop-active video frames by the prop fingerprint model, and are used to represent the image features of the prop in the prop-active video frames; and an activation judgment unit, configured to determine whether the prop in the target video frame is active based on the first prop fingerprint vector and the second prop fingerprint vector.

[0008] Fifthly, embodiments of this specification provide a model training apparatus, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method described in the first aspect above.

[0009] Sixthly, embodiments of this specification provide a prop activation determination device, comprising: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method described in the second aspect above.

[0010] In a seventh aspect, embodiments of this specification provide a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the steps of the method described in the first aspect above.

[0011] Eighthly, embodiments of this specification provide a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the steps of the method described in the second aspect above.

[0012] In this embodiment, a prop fingerprint model is trained. This model generates a first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective. The first prop fingerprint vector represents the image features of the prop in the target video frame. Therefore, this embodiment utilizes the prop fingerprint model to generate a first prop fingerprint vector corresponding to the target video frame, and uses this first prop fingerprint vector to determine whether the prop in the target video frame is effective. Compared to existing technologies that rely on user-executed prop addition operations and determine prop effectiveness only after detecting the prop addition operation, this embodiment considers the situation where the user performs the prop addition operation but the prop is not effective. It can accurately determine whether the prop added in the video is effective, improving the accuracy of determining whether the prop in the video is effective. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A schematic flowchart illustrating a model training method provided in one embodiment of this specification;

[0015] Figure 2 A schematic diagram of a training sample in the form of a triplet provided in one embodiment of this specification;

[0016] Figure 3 This is a schematic diagram illustrating the process of constructing training samples in the form of triples, provided as an embodiment of this specification.

[0017] Figure 4 A flowchart illustrating the process of constructing training samples in the form of triples, provided for another embodiment of this specification;

[0018] Figure 5 A flowchart illustrating a method for determining the effectiveness of an item as provided in one embodiment of this specification;

[0019] Figure 6 A flowchart illustrating a method for determining the effectiveness of an item, provided in another embodiment of this specification;

[0020] Figure 7 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification;

[0021] Figure 8 This is a schematic diagram of the structure of a prop activation determination device provided in one embodiment of this specification;

[0022] Figure 9 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0024] This specification provides a model training method and a prop effectiveness determination method, which can accurately determine whether the props added to the video are effective.

[0025] Figure 1 This is a flowchart illustrating a model training method provided in one embodiment of this specification, as shown below. Figure 1 As shown, the process includes:

[0026] Step S102: Construct training samples in the form of triples; the training samples include a baseline video frame, a positive example video frame, and a negative example video frame; the baseline video frame and the positive example video frame have the same active props; the baseline video frame and the negative example video frame have different active props.

[0027] Step S104: Using training samples in the form of triples and a preset loss function, train a pre-built neural network structure; the loss function is used to make the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the positive example video frame less than a first preset distance, and to make the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the negative example video frame greater than a second preset distance.

[0028] Step S106: The trained neural network structure is used as the prop fingerprint model; the prop fingerprint model is used to generate the first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0029] In this embodiment, a prop fingerprint model is trained. This model generates a first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective. The first prop fingerprint vector represents the image features of the prop in the target video frame. Therefore, this embodiment utilizes the prop fingerprint model to generate a first prop fingerprint vector corresponding to the target video frame, and uses this first prop fingerprint vector to determine whether the prop in the target video frame is effective. Compared to existing technologies that rely on user-executed prop addition operations and determine prop effectiveness only after detecting the prop addition operation, this embodiment considers the situation where the user performs the prop addition operation but the prop is not effective. It can accurately determine whether the prop added in the video is effective, improving the accuracy of determining whether the prop in the video is effective.

[0030] The model training method in this embodiment can be executed by a device used for model training, which can be a backend server or a server cluster. Here, we briefly introduce the role of the prop fingerprint model. When applying the prop fingerprint model, video frames can be input into it. The prop fingerprint model processes the input video frames, generating prop fingerprint vectors corresponding to the input video frames. These prop fingerprint vectors represent the image features of the props in the input video frames. For two video frames with the same prop added and active, the generated prop fingerprint vectors are relatively close, and the distance between the prop fingerprint vectors is less than or equal to a first preset distance. For two video frames with different props added and active, the generated prop fingerprint vectors are relatively far apart, and the distance between the prop fingerprint vectors is greater than or equal to a second preset distance. The second preset distance can be greater than the first preset distance. The props mentioned in this embodiment include, but are not limited to, props such as headwear, backgrounds, borders, and clothing added by users during image processing such as photo editing and video creation. Images with added props have certain display effects. The following details each process of the model training method.

[0031] In step S102 above, training samples in the form of triples are constructed. These training samples include a baseline video frame, positive example video frames, and negative example video frames. The baseline video frame and the positive example video frames have the same active props. The baseline video frame and the negative example video frames have different active props. In this embodiment, metric learning is used to train the prop fingerprint model; therefore, it is necessary to first construct the positive and negative example triples required for metric learning, that is, to construct training samples in the form of triples. Figure 2 A schematic diagram of a training sample in the form of a triplet provided in one embodiment of this specification, as shown below. Figure 2As shown, the training samples in triplet form include a baseline video frame anchor (a) in the figure, positive video frames (p) in the figure, and negative video frames (n) in the figure. The baseline video frame anchor and the positive video frames are video frames using the same prop, while the negative video frames are video frames using different props. Figure 2 As shown, the prop fingerprint model should make the prop fingerprint vectors obtained by the reference video frame and the positive example video frame as close in space as possible, while making the prop fingerprint vectors obtained by the reference video frame and the negative example video frame as far apart in space, and making the prop fingerprint vectors obtained by the positive example video frame and the negative example video frame as far apart in space.

[0032] In one embodiment, training samples in the form of triples can be constructed using the logging data of a pre-defined video application. Figure 3 This is a schematic diagram illustrating the process of constructing training samples in the form of triples according to an embodiment of this specification, as shown below. Figure 3 As shown, the process includes:

[0033] Step S302: Obtain the tracking log of the preset video application, and filter out multiple first videos in which the props are effective from the tracking log; the user usage popularity of the props in the first videos is higher than the preset popularity.

[0034] Step S304: Extract a reference video frame, a positive example video frame, and a negative example video frame from multiple first videos; wherein, the reference video frame and the positive example video frame are first video frames with the same effective props and background content that meets the preset similarity requirements, or, the reference video frame and the negative example video frame are first video frames with the same effective props but background content that does not meet the preset similarity requirements;

[0035] Step S306: Construct training samples in the form of triplets based on the extracted baseline video frames, positive example video frames, and negative example video frames.

[0036] The preset video application can be any image processing application. There can be one or more preset video applications. In step S302, the tracking log of the preset video application is first obtained. The tracking log stores multiple videos and records marker information indicating whether the props in the videos are effective. Based on this marker information, it can be determined whether the props in the video are effective. When users process images using the preset video application, they will select props to add to the image according to their preferences. The user popularity of a prop can be determined based on the number of times it is used by the user. For example, the user popularity of a prop can be calculated according to a preset formula based on the number of times it is used by the user; the higher the number of times a prop is used, the higher its user popularity. In this step, when selecting the first video, multiple videos where the props are effective are first selected from the tracking log based on the marker information recorded in the tracking log. Then, multiple videos with a user popularity of props higher than a preset popularity are selected as the first video. This ensures that the props in the selected first video are frequently used, high-popularity props, making the generated training samples adaptable to user needs and targeted.

[0037] In step S304, a reference video frame, a positive example video frame, and a negative example video frame are extracted from multiple first videos. As described above, the reference video frame and the positive example video frame have the same active prop. In this step, by extracting video frames, two first video frames with the same active prop and background content that meets the preset similarity requirements can be extracted as the reference video frame and the positive example video frame. Considering that users usually use the same prop in a video and the background content of a video remains largely unchanged, the reference video frame and the positive example video frame can be different video frames in the same first video. In a specific embodiment, after determining that a first video frame with a prop is the reference video frame, according to the above analysis, different video frames in the first video containing the reference video frame can be selected as positive example video frames. In this case, the reference video frame and the positive example video frame constitute a positive sample that is easier to learn.

[0038] Alternatively, two first video frames with the same active prop but whose background content does not meet the preset similarity requirement can be extracted as the reference video frame and the positive example video frame. Considering that the background content of the first videos of different users are not very similar, but different users may use the same prop, in this case, the reference video frame and the positive example video frame can be video frames from different first videos that use the same prop. In a specific embodiment, after determining that a first video frame with a prop is the reference video frame, according to the above analysis, video frames from different first videos that use the same prop as the reference video frame can be selected as positive example video frames. In this case, the reference video frame and the positive example video frame constitute positive samples that are more difficult to learn.

[0039] As described above, the reference video frame and the negative example video frame have different active props. In this step, by extracting video frames, two first video frames with different active props and background content that do not meet the preset similarity requirements can be extracted as the reference video frame and the negative example video frame. Considering that the background content of the first videos of different users is not very similar, and different users may use different props, in this case, the reference video frame and the negative example video frame can be video frames from different first videos, which use different props. In a specific embodiment, after determining that a first video frame with a prop is the reference video frame, according to the above analysis, video frames from different first videos that use different props than the reference video frame can be selected as negative example video frames. In this case, the reference video frame and the negative example video frame constitute a negative sample that is easier to learn.

[0040] It is understandable that, since the background content of the first videos of different users is not very similar, even if different users use different props, it is difficult to extract two first video frames as the reference video frame and the negative example video frame by extracting video frames from the first video in this step, even if the effective props are different but the background content meets the preset similarity requirements.

[0041] In step S306 above, the extracted baseline video frames, positive example video frames, and negative example video frames are grouped into training samples in the form of triplets. Each training sample includes one baseline video frame, one positive example video frame, and one negative example video frame. In this embodiment, the training samples are constructed based on the logging data, which has the advantages of low construction cost and easy sample acquisition.

[0042] In another embodiment, training samples in the form of triples can also be constructed by automatically adding props. Figure 4 This is a flowchart illustrating the process of constructing training samples in the form of triples, as provided in another embodiment of this specification. Figure 4 As shown, the process includes:

[0043] Step S402: Obtain multiple second videos that have not used props, and use an automated script to add props to the multiple second videos;

[0044] Step S404: Extract a reference video frame, a positive example video frame, and a negative example video frame from the multiple second videos after adding props; wherein, the reference video frame and the positive example video frame are second video frames with the same effective props and background content that meets the preset similarity requirements, or, second video frames with the same effective props but background content that does not meet the preset similarity requirements; the reference video frame and the negative example video frame are second video frames with different effective props and background content that does not meet the preset similarity requirements, or, second video frames with different effective props but background content that meets the preset similarity requirements;

[0045] Step S406: Construct training samples in the form of triplets based on the extracted baseline video frames, positive example video frames, and negative example video frames.

[0046] In step S402 above, multiple second videos without used props can be retrieved from the background video database or from the network, and each prop in a pre-established prop library can be retrieved. Each prop in the prop library is the prop to be followed. An automated script is used to randomly add each prop from the prop library to the multiple second videos. When adding props, for a single second video, the same second video can be copied into multiple copies, and different props can be added to each copy. For each prop, it can be added to one second video or multiple second videos.

[0047] In step S404 above, a reference video frame, a positive example video frame, and a negative example video frame are extracted from multiple second videos after adding props. As described above, the reference video frame and the positive example video frame have the same effective props. In this step, by extracting video frames, two second video frames with the same effective props and background content that meets the preset similarity requirements can be extracted as the reference video frame and the positive example video frame. Considering that the background content of a video remains largely unchanged, in this case, the reference video frame and the positive example video frame can be different video frames in the same second video. In a specific embodiment, after determining that a second video frame with props is the reference video frame, according to the above analysis, different video frames in the second video containing the reference video frame can be selected as positive example video frames. In this case, the reference video frame and the positive example video frame constitute a positive sample that is easier to learn.

[0048] Alternatively, two second video frames with the same active props but background content that does not meet the preset similarity requirement can be extracted as the baseline video frame and the positive example video frame. Considering that the background content of different second videos is not very similar, but the same props can be added to different second videos, the baseline video frame and the positive example video frame can be video frames from different second videos with the same props added. In a specific embodiment, after determining that a second video frame with props is the baseline video frame, according to the above analysis, video frames from different second videos with the same props added to the baseline video frame can be selected as positive example video frames. In this case, the baseline video frame and the positive example video frame constitute positive samples that are more difficult to learn.

[0049] As described above, the reference video frame and the negative example video frame have different active props. In this step, by extracting video frames, two second video frames with different active props and background content that does not meet the preset similarity requirement can be extracted as the reference video frame and the negative example video frame. Considering that the background content of different second videos is not very similar, and different props can be added to different second videos, the reference video frame and the negative example video frame can be video frames from different second videos that use different props. In a specific embodiment, after determining that a first video frame with a prop is the reference video frame, according to the above analysis, video frames from different second videos that use different props than the reference video frame can be selected as negative example video frames. In this case, the reference video frame and the negative example video frame constitute a negative sample that is easier to learn.

[0050] Alternatively, two second video frames with different effective props but background content that meets a preset similarity requirement can be extracted as the reference video frame and the negative example video frame. Considering that the same second video can be copied multiple times and different props can be added to each copy, in this case, the reference video frame and the negative example video frame can be located in the copied second videos, with different props used in different copied second videos. In a specific embodiment, after determining that a first video frame with a prop is the reference video frame, according to the above analysis, the same second video frame using a different prop from the reference video frame can be selected as the negative example video frame. In this case, the reference video frame and the negative example video frame constitute a negative sample that is more difficult to learn.

[0051] In step S406 above, the extracted baseline video frames, positive example video frames, and negative example video frames are grouped into training samples in the form of triplets. Each training sample includes one baseline video frame, one positive example video frame, and one negative example video frame. In this embodiment, the training samples are constructed based on the method of adding props using an automated script, which has the following advantages: 1) It can ensure that the props in the second video are effective; 2) It can construct difficult-to-learn negative example samples with different effective props but background content that meets the preset similarity requirements, thus enriching the diversity of the samples.

[0052] In a specific embodiment, it can be first through Figure 3 A batch of training samples is obtained by constructing the model in the method described above, and then... Figure 4 Another batch of training samples was constructed using the method described above, and the model parameters were optimized after the initial training of the model converged.

[0053] Figure 1 In step S104, the pre-built neural network structure is trained using the constructed training samples in the form of triples and a preset loss function. This loss function is used to ensure that the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the positive example video frame is less than or equal to a first preset distance, and to ensure that the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the negative example video frame is greater than or equal to a second preset distance.

[0054] In one embodiment, the pre-built neural network structure can be a ResNet50 network structure. The loss function can be expressed by the following formula (1).

[0055]

[0056] As shown in formula (1), where v a v represents the prop fingerprint vector corresponding to the reference video frame. p v represents the prop fingerprint vector corresponding to the positive example video frame. n Dist(v) represents the prop fingerprint vector corresponding to the negative example video frame. a ,v p d(v) represents the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the positive example video frame. a ,v n ) represents the distance between the prop fingerprint vector corresponding to the baseline video frame and the prop fingerprint vector corresponding to the negative example video frame, and margin is a parameter preset by the model. As shown in formula (1), when the loss function LOSS is 0, dist(v a ,v p ) is 0, and d(v) a ,v nThe distance between the prop fingerprint vectors of the baseline video frame and the positive example video frame is equal to or greater than the margin. Therefore, based on this loss function, the first preset distance is 0, and the second preset distance is the margin. This loss function ensures that the distance between the prop fingerprint vectors of the baseline video frame and the positive example video frame is 0, and that the distance between the prop fingerprint vectors of the baseline video frame and the negative example video frame is greater than or equal to the margin, thus satisfying the model training requirements.

[0057] Of course, the first preset distance does not have to be 0. The loss function can be designed according to the requirements, and the first preset distance and the second preset distance can be set, as long as the first preset distance is less than the second preset distance.

[0058] Figure 1 In step S106, the trained neural network structure is used as the prop fingerprint model. Thus, after the prop fingerprint model is trained, it can process the input video frames, generate the prop fingerprint vectors corresponding to the input video frames, and ensure that for two video frames with the same props, the generated prop fingerprint vectors are close to each other, and for two video frames with different props, the generated prop fingerprint vectors are far apart.

[0059] In this embodiment, the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to the target video frame, in order to determine whether the prop in the target video frame is effective. The first prop fingerprint vector is used to represent the image features of the prop in the target video frame. Specifically, the target video frame is input into the prop fingerprint model, and the prop fingerprint model processes the target video frame to obtain the corresponding first prop fingerprint vector.

[0060] In this embodiment, the prop fingerprint model is also used to establish a prop fingerprint vector library. The prop fingerprint vector library stores multiple second prop fingerprint vectors. These second prop fingerprint vectors are obtained by processing pre-acquired prop activation video frames using the prop fingerprint model, and are used to represent the image features of the props in the activation video frames.

[0061] Specifically, a prop library is pre-established, and each prop in the library is a prop of interest. Multiple video frames without added props can be acquired, and props from the prop library can be added to these video frames to obtain prop-activated video frames. For the same prop, multiple prop-activated video frames can be generated. Alternatively, for each prop, at least one prop-activated video is obtained from the tracking log, and prop-activated video frames are extracted from these videos. These prop-activated video frames are input into a prop fingerprint model, which generates a second prop fingerprint vector corresponding to each prop-activated video frame. The various second prop fingerprint vectors are combined to obtain a prop fingerprint vector library. The second prop fingerprint vectors in the prop fingerprint vector library can be combined with the first prop fingerprint vectors mentioned above to determine whether the prop in the target video frame is effective.

[0062] The above describes the specific training process and function of the prop fingerprint model. The following describes the specific process of determining whether a prop is effective based on the prop fingerprint model.

[0063] Figure 5 This is a flowchart illustrating a method for determining the effectiveness of an item according to an embodiment of this specification, as shown below. Figure 5 As shown, the process includes:

[0064] Step S502: Obtain the target video frame for determining whether the prop is effective; process the target video frame using the prop fingerprint model trained above to obtain the first prop fingerprint vector corresponding to the target video frame; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0065] Step S504: Obtain a pre-established prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vectors are obtained by processing the pre-acquired prop activation video frames by the prop fingerprint model, and are used to represent the image features of the props in the prop activation video frames.

[0066] Step S506: Determine whether the prop in the target video frame is effective based on the first prop fingerprint vector and the second prop fingerprint vector.

[0067] In this embodiment, a prop fingerprint model is trained. This model generates a first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective. The first prop fingerprint vector represents the image features of the prop in the target video frame. Therefore, this embodiment utilizes the prop fingerprint model to generate a first prop fingerprint vector corresponding to the target video frame, and uses this first prop fingerprint vector to determine whether the prop in the target video frame is effective. Compared to existing technologies that rely on user-executed prop addition operations and determine prop effectiveness only after detecting the prop addition operation, this embodiment considers the situation where the user performs the prop addition operation but the prop is not effective. It can accurately determine whether the prop added in the video is effective, improving the accuracy of determining whether the prop in the video is effective.

[0068] Figure 5 The method described can be executed by the device used to train the prop fingerprint model, or by other devices, such as a device specifically designed for prop identification.

[0069] In step S502 above, the target video frame is the video frame from which the detection of whether the prop is effective is performed. The target video frame is input into the prop fingerprint model trained above, and the first prop fingerprint vector corresponding to the target video frame output by the prop fingerprint model is obtained. The first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0070] In step S504 above, a pre-established item fingerprint vector library is obtained. The item fingerprint vector library can be constructed in the following way:

[0071] (1) For each prop in the preset prop library, obtain the corresponding prop activation video frame;

[0072] (2) Input the prop activation video frame into the prop fingerprint model for processing, and obtain the second prop fingerprint vector corresponding to the prop activation video frame output by the prop fingerprint model;

[0073] (3) Use the second item fingerprint vector to construct an item fingerprint vector library.

[0074] The process of building the prop fingerprint vector library can be referred to the previous description and will not be repeated here. The construction process of the prop fingerprint vector library can be performed by the device used to train the prop fingerprint model, or by... Figure 5 The corresponding execution device can be used, or it can be executed by other devices.

[0075] In step S506 above, the effectiveness of the prop in the target video frame is determined based on the first prop fingerprint vector and the second prop fingerprint vector. In one embodiment, this step specifically involves:

[0076] (1) Calculate the distance between the first item fingerprint vector and each second item fingerprint vector, and select a preset number of target second item fingerprint vectors that are closest to the first item fingerprint vector from each second item fingerprint vector based on the distance.

[0077] (2) Obtain the first item identifier of the item in the target video frame, obtain the target item identifier corresponding to the fingerprint vector of each target second item, and determine whether the item in the target video frame is effective based on the first item identifier and the target item identifier; wherein, the target item identifier is the identifier of the item in the video frame where the item is effective, corresponding to the fingerprint vector of the target second item.

[0078] First, calculate the distance between the first item fingerprint vector and each of the second item fingerprint vectors. Based on this distance, select a preset number of target second item fingerprint vectors that are closest to the first item fingerprint vector from among the second item fingerprint vectors. For example, select 30 target second item fingerprint vectors that are closest to the first item fingerprint vector from among the second item fingerprint vectors based on this distance.

[0079] Then, the first item identifier of the item in the target video frame is obtained, which is the identifier of the item used to determine whether it is effective. Furthermore, the target item identifier corresponding to each target second item fingerprint vector is obtained, which is the identifier of the item in the video frame where the item is effective, corresponding to the target second item fingerprint vector.

[0080] Finally, based on the first prop identifier and the target prop identifier, it is determined whether the prop in the target video frame is effective. In one embodiment, this action specifically involves:

[0081] (21) Determine whether the first item icon exists among the target item icons;

[0082] (22) If it exists, determine that the prop in the target video frame is effective; if it does not exist, determine that the prop in the target video frame is not effective.

[0083] Search for the first item icon among the various target item icons. If found, it means that among the 30 target second item fingerprint vectors closest to the first item fingerprint vector, there is an item from the target video frame. Therefore, the item in the target video frame is confirmed to be effective. Conversely, if not found, it means that among the 30 target second item fingerprint vectors closest to the first item fingerprint vector, there is no item from the target video frame. Therefore, the item in the target video frame is confirmed to be ineffective.

[0084] In another embodiment, based on the first prop identifier and the target prop identifier, it is determined whether the prop in the target video frame is effective. This action specifically involves:

[0085] (23) Vote for each target item identifier, and the voting result is used to indicate whether the corresponding target item identifier is the same as the first item identifier;

[0086] (24) If the number of preset voting results in the voting results exceeds the preset proportion, the prop in the target video frame is determined to be effective; otherwise, the prop in the target video frame is determined to be ineffective. The preset voting results are used to indicate that the corresponding target prop identifier is the same as the first prop identifier.

[0087] First, each target item identifier is voted on individually. The voting result indicates whether the corresponding target item identifier is the same as the first item identifier. The specific voting process can be as follows: compare whether the target item identifier is the same as the first item identifier, and generate the voting result based on the comparison result.

[0088] Next, the preset voting results are searched among the various voting results. Preset voting results indicate that the corresponding target item identifier is the same as the first item identifier. The total number of all voting results is determined; this number is equal to the number of target item identifiers. The number of preset voting results among all voting results is determined; this number is necessarily less than the total number of voting results. The proportion of preset voting results to the total number of voting results is calculated. If this proportion exceeds a preset proportion, such as exceeding 50%, the item in the target video frame is determined to be effective. If this proportion does not exceed the preset proportion, such as not exceeding 50%, the item in the target video frame is determined to be ineffective.

[0089] In this embodiment, by voting and judging the percentage of preset voting results, the accuracy of the result of judging whether the prop is effective can be guaranteed.

[0090] In one embodiment, obtaining the target video frame for determining whether the prop to be detected is effective specifically involves: obtaining the target video for determining whether the prop to be detected is effective, and selecting target video frames from the target video. Accordingly, after determining whether the prop in the target video frame is effective, the method further includes: determining whether the prop in the target video is effective based on the determination result of the target video frame.

[0091] Figure 6 This is a flowchart illustrating another embodiment of the method for determining the effectiveness of an item provided in this specification, as shown below. Figure 6 As shown, the process includes:

[0092] Step S602: Obtain the target video to determine whether the prop to be tested is effective;

[0093] Step S604: Select target video frames from the target video, process the target video frames using the prop fingerprint model trained above, and obtain the first prop fingerprint vector corresponding to the target video frame.

[0094] Step S606: Obtain the pre-established item fingerprint vector library; the item fingerprint vector library stores multiple second item fingerprint vectors;

[0095] Step S608: Determine whether the prop in the target video frame is effective based on the first prop fingerprint vector and the second prop fingerprint vector;

[0096] Step S610: Based on the judgment result of the target video frame, determine whether the props in the target video are effective.

[0097] In step S602 above, the target video for determining whether the item to be tested is effective is obtained. In step S604 above, the target video is decoded and frames are extracted to obtain the target video frame. The target video can be video data with an item effectiveness marker in the log. This embodiment further determines whether the item in the video data with the item effectiveness marker is actually effective. The process of steps S604-S608 is detailed in the above description and will not be repeated here. In step S610, based on the judgment result of the target video frame, it is determined whether the item in the target video is effective. Specifically: if the judgment result of the target video frame indicates that the item in the target video frame is effective, then the item in the target video is determined to be effective; if the judgment result of the target video frame indicates that the item in the target video frame is not effective, then the item in the target video is determined to be not effective.

[0098] The number of target video frames can be multiple or a single frame, and the target video frames can be randomly selected from the target video. If the judgment result of any target video frame indicates that the prop in the target video frame is effective, then the prop in the target video is determined to be effective; otherwise, the prop in the target video is determined to be ineffective.

[0099] As can be seen, in this embodiment, it is possible not only to determine whether the props in a certain frame of video are effective, but also to determine whether the props in a whole video are effective.

[0100] In summary, by using the above model training methods and prop effectiveness determination methods, the prop fingerprint model can be used to determine whether props in target video frames are effective by generating prop fingerprint vectors. The accuracy is high, and the video frames after determination can be tagged with prop effectiveness to improve the accuracy of tagging.

[0101] Figure 7 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification, as shown below. Figure 7 As shown, the device includes:

[0102] The sample construction unit 71 is used to construct training samples in the form of triples; the training samples include a baseline video frame, positive example video frames, and negative example video frames; the baseline video frame and the positive example video frame have the same active props; the baseline video frame and the negative example video frame have different active props.

[0103] The model training unit 72 is used to train a pre-built neural network structure using the training samples in the form of triples and a preset loss function; the loss function is used to make the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the positive example video frame less than a first preset distance, and to make the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the negative example video frame greater than a second preset distance.

[0104] The model generation unit 73 is used to use the trained neural network structure as a prop fingerprint model; the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0105] Optionally, the sample construction unit 71 is specifically used for: obtaining the timing logs of a preset video application, and filtering multiple first videos from the timing logs in which the props are effective; the user usage popularity of the effective props in the first videos is higher than a preset popularity; extracting a benchmark video frame, a positive example video frame, and a negative example video frame from the multiple first videos; wherein, the benchmark video frame and the positive example video frame are first video frames in which the effective props are the same and the background content meets a preset similarity requirement, or, are first video frames in which the effective props are the same but the background content does not meet the preset similarity requirement; the benchmark video frame and the negative example video frame are first video frames in which the effective props are different and the background content does not meet the preset similarity requirement; and constructing training samples in the form of triplets based on the extracted benchmark video frames, positive example video frames, and negative example video frames.

[0106] Optionally, the sample construction unit 71 is specifically used for: acquiring multiple second videos without props, and adding props to the multiple second videos using an automated script; extracting a baseline video frame, a positive example video frame, and a negative example video frame from the multiple second videos after adding props; wherein the baseline video frame and the positive example video frame are second video frames with the same effective props and background content that meets a preset similarity requirement, or, second video frames with the same effective props but background content that does not meet the preset similarity requirement; the baseline video frame and the negative example video frame are second video frames with different effective props and background content that does not meet the preset similarity requirement, or, second video frames with different effective props but background content that meets the preset similarity requirement; and constructing training samples in the form of triplets based on the extracted baseline video frames, positive example video frames, and negative example video frames.

[0107] Optionally, the prop fingerprint model is further used to establish a prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vector is obtained by the prop fingerprint model processing a pre-acquired prop-effective video frame, and is used to represent the image features of the prop in the prop-effective video frame; the first prop fingerprint vector is combined with the second prop fingerprint vector to determine whether the prop in the target video frame is effective.

[0108] The model training device in this embodiment can realize each process of the aforementioned model training method and achieve the same function and effect, which will not be repeated here.

[0109] Figure 8 This is a schematic diagram of the structure of a prop activation determination device provided in one embodiment of this specification, as shown below. Figure 8 As shown, the device includes:

[0110] The first acquisition unit 81 is used to acquire the target video frame of whether the prop to be detected is effective, and to process the target video frame using the prop fingerprint model trained as described above to obtain the first prop fingerprint vector corresponding to the target video frame; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0111] The second acquisition unit 82 is used to acquire a pre-established prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vector is obtained by the prop fingerprint model processing the pre-acquired prop effective video frames, and is used to represent the image features of the props in the prop effective video frames;

[0112] The effectiveness determination unit 83 is used to determine whether the prop in the target video frame is effective based on the first prop fingerprint vector and the second prop fingerprint vector.

[0113] Optionally, the effectiveness determination unit 83 is specifically used for: calculating the distance between the first item fingerprint vector and each of the second item fingerprint vectors; selecting a preset number of target second item fingerprint vectors that are closest to the first item fingerprint vector from among the second item fingerprint vectors according to the distance; obtaining the first item identifier of the item in the target video frame; obtaining the target item identifier corresponding to each of the target second item fingerprint vectors; and determining whether the item in the target video frame is effective based on the first item identifier and the target item identifier; wherein, the target item identifier is the identifier of the item in the video frame where the item is effective, corresponding to the target second item fingerprint vector.

[0114] Optionally, the effectiveness determination unit 83 is further configured to: vote on each of the target prop identifiers respectively, and the voting result is used to indicate whether the corresponding target prop identifier is the same as the first prop identifier; if the number of preset voting results in the voting results exceeds a preset proportion, then the prop in the target video frame is determined to be effective, otherwise, the prop in the target video frame is determined to be ineffective; wherein, the preset voting result is used to indicate that the corresponding target prop identifier is the same as the first prop identifier.

[0115] Optionally, the first acquisition unit 81 is specifically used to: acquire a target video of whether the prop to be detected is effective, and select the target video frame from the target video; the device further includes: a result judgment unit, used to determine whether the prop in the target video is effective based on the judgment result of the target video frame after judging whether the prop in the target video frame is effective.

[0116] Optionally, the result judgment unit is specifically used to: if the judgment result of the target video frame indicates that the prop in the target video frame is effective, then determine that the prop in the target video is effective; if the judgment result of the target video frame indicates that the prop in the target video frame is not effective, then determine that the prop in the target video is not effective.

[0117] Optionally, the prop fingerprint vector library is constructed in the following way: for a prop in the preset prop library, obtain the prop activation video frame corresponding to the prop; input the prop activation video frame into the prop fingerprint model for processing, and obtain the second prop fingerprint vector corresponding to the prop activation video frame output by the prop fingerprint model; use the second prop fingerprint vector to construct the prop fingerprint vector library.

[0118] The item activation determination device in this embodiment can realize all the processes of the aforementioned item activation determination method and achieve the same function and effect, which will not be repeated here.

[0119] This specification also provides an electronic device according to one embodiment. Figure 9 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification, such as... Figure 9 As shown, electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 1301 and memories 1302, with the memory 1302 storing one or more application programs or data. The memory 1302 can be temporary or persistent storage. The application programs stored in the memory 1302 may include one or more modules (not shown), each module including a series of computer-executable instructions within the electronic device. Furthermore, the processor 1301 may be configured to communicate with the memory 1302, executing the series of computer-executable instructions stored in the memory 1302 on the electronic device. The electronic device may also include one or more power supplies 1303, one or more wired or wireless network interfaces 1304, one or more input or output interfaces 1305, one or more keyboards 1306, etc.

[0120] In one specific embodiment, the electronic device is a model training device, including a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following process:

[0121] Construct training samples in the form of triplets; the training samples include a baseline video frame, positive example video frames, and negative example video frames; the baseline video frame and the positive example video frame have the same active props; the baseline video frame and the negative example video frame have different active props.

[0122] Using the training samples in the form of triples and a preset loss function, a pre-built neural network structure is trained; the loss function is used to make the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the positive example video frame less than or equal to a first preset distance, and to make the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the negative example video frame greater than or equal to a second preset distance.

[0123] The trained neural network structure is used as a prop fingerprint model; the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0124] Optionally, when the computer-executable instructions are executed, they construct training samples in the form of triplets, including: obtaining a logging log of a preset video application; filtering multiple first videos from the logging log where props are active; the user usage popularity of the active props in the first videos is higher than a preset popularity; extracting a baseline video frame, a positive example video frame, and a negative example video frame from the multiple first videos; wherein the baseline video frame and the positive example video frame are first video frames with the same active props and background content that meets a preset similarity requirement, or, are first video frames with the same active props but background content that does not meet the preset similarity requirement; the baseline video frame and the negative example video frame are first video frames with different active props and background content that does not meet the preset similarity requirement; and constructing training samples in the form of triplets based on the extracted baseline video frames, positive example video frames, and negative example video frames.

[0125] Optionally, when the computer-executable instructions are executed, they construct training samples in the form of triplets, including: acquiring multiple second videos without props; adding props to the multiple second videos using an automated script; extracting a baseline video frame, a positive example video frame, and a negative example video frame from the multiple second videos after adding props; wherein the baseline video frame and the positive example video frame are second video frames with the same effective props and background content that meets a preset similarity requirement, or second video frames with the same effective props but background content that does not meet the preset similarity requirement; the baseline video frame and the negative example video frame are second video frames with different effective props and background content that does not meet the preset similarity requirement, or second video frames with different effective props but background content that meets the preset similarity requirement; and constructing training samples in the form of triplets based on the extracted baseline video frames, positive example video frames, and negative example video frames.

[0126] Optionally, when the computer-executable instructions are executed, the prop fingerprint model is further used to establish a prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vectors are obtained by the prop fingerprint model processing pre-acquired prop-effective video frames, and are used to represent the image features of the props in the prop-effective video frames; the first prop fingerprint vectors are combined with the second prop fingerprint vectors to determine whether the props in the target video frame are effective.

[0127] The model training device in this embodiment can implement all the processes of the aforementioned model training method and achieve the same functions and effects, which will not be repeated here.

[0128] In another specific embodiment, the electronic device is a prop activation determination device, including a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following process:

[0129] The target video frame for determining whether a prop is effective is obtained. The target video frame is then processed using the prop fingerprint model trained as described above to obtain the first prop fingerprint vector corresponding to the target video frame. The first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0130] Obtain a pre-established prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vectors are obtained by the prop fingerprint model processing the pre-obtained prop activation video frames, and are used to represent the image features of the props in the prop activation video frames;

[0131] Based on the first prop fingerprint vector and the second prop fingerprint vector, determine whether the prop in the target video frame is effective.

[0132] Optionally, when the computer-executable instruction is executed, determining whether a prop in the target video frame is effective based on the first prop fingerprint vector and the second prop fingerprint vector includes: calculating the distance between the first prop fingerprint vector and each of the second prop fingerprint vectors; selecting a preset number of target second prop fingerprint vectors that are closest to the first prop fingerprint vector from among the second prop fingerprint vectors based on the distance; obtaining a first prop identifier of the prop in the target video frame; obtaining a target prop identifier corresponding to each of the target second prop fingerprint vectors; and determining whether a prop in the target video frame is effective based on the first prop identifier and the target prop identifier; wherein, the target prop identifier is the identifier of the prop in the video frame where the prop is effective, corresponding to the target second prop fingerprint vector.

[0133] Optionally, when the computer-executable instruction is executed, determining whether the prop in the target video frame is effective based on the first prop identifier and the target prop identifier includes: voting on each of the target prop identifiers, with the voting result indicating whether the corresponding target prop identifier is the same as the first prop identifier; if the proportion of preset voting results in the voting results exceeds a preset ratio, then the prop in the target video frame is determined to be effective; otherwise, the prop in the target video frame is determined to be ineffective; wherein, the preset voting result indicates that the corresponding target prop identifier is the same as the first prop identifier.

[0134] Optionally, when the computer-executable instruction is executed, acquiring a target video frame indicating whether the prop to be detected is effective includes: acquiring a target video indicating whether the prop to be detected is effective, and selecting the target video frame from the target video; after determining whether the prop in the target video frame is effective, the instruction further includes: determining whether the prop in the target video is effective based on the determination result of the target video frame.

[0135] Optionally, when the computer-executable instruction is executed, it determines whether the props in the target video are effective based on the judgment result of the target video frame, including: if the judgment result of the target video frame indicates that the props in the target video frame are effective, then it is determined that the props in the target video are effective; if the judgment result of the target video frame indicates that the props in the target video frame are not effective, then it is determined that the props in the target video are not effective.

[0136] Optionally, when the computer-executable instructions are executed, the item fingerprint vector library is constructed in the following manner: for an item in the preset item library, obtain the item activation video frame corresponding to the item; input the item activation video frame into the item fingerprint model for processing, and obtain the second item fingerprint vector corresponding to the item activation video frame output by the item fingerprint model; and construct the item fingerprint vector library using the second item fingerprint vector.

[0137] The item activation determination device in this embodiment can implement all the processes of the aforementioned item activation determination method and achieve the same function and effect, which will not be repeated here.

[0138] Another embodiment of this specification provides a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the following process:

[0139] Construct training samples in the form of triplets; the training samples include a baseline video frame, positive example video frames, and negative example video frames; the baseline video frame and the positive example video frame have the same active props; the baseline video frame and the negative example video frame have different active props.

[0140] Using the training samples in the form of triples and a preset loss function, a pre-built neural network structure is trained; the loss function is used to make the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the positive example video frame less than or equal to a first preset distance, and to make the distance between the prop fingerprint vector corresponding to the reference video frame and the prop fingerprint vector corresponding to the negative example video frame greater than or equal to a second preset distance.

[0141] The trained neural network structure is used as a prop fingerprint model; the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0142] Optionally, when the computer-executable instructions are executed by the processor, the training samples in the form of triplets are constructed, including: obtaining the timing logs of a preset video application; filtering multiple first videos from the timing logs in which props are effective; the user usage popularity of the effective props in the first videos is higher than a preset popularity; extracting a baseline video frame, a positive example video frame, and a negative example video frame from the multiple first videos; wherein the baseline video frame and the positive example video frame are first video frames in which the effective props are the same and the background content meets a preset similarity requirement, or, are first video frames in which the effective props are the same but the background content does not meet the preset similarity requirement; the baseline video frame and the negative example video frame are first video frames in which the effective props are different and the background content does not meet the preset similarity requirement; and constructing training samples in the form of triplets based on the extracted baseline video frames, positive example video frames, and negative example video frames.

[0143] Optionally, when the computer-executable instructions are executed by the processor, the training samples in the form of triplets are constructed, including: acquiring multiple second videos without props; adding props to the multiple second videos using an automated script; extracting a baseline video frame, a positive example video frame, and a negative example video frame from the multiple second videos after adding props; wherein the baseline video frame and the positive example video frame are second video frames with the same effective props and background content that meets a preset similarity requirement, or, second video frames with the same effective props but background content that does not meet the preset similarity requirement; the baseline video frame and the negative example video frame are second video frames with different effective props and background content that does not meet the preset similarity requirement, or, second video frames with different effective props but background content that meets the preset similarity requirement; and constructing training samples in the form of triplets based on the extracted baseline video frames, positive example video frames, and negative example video frames.

[0144] Optionally, when the computer-executable instructions are executed by the processor, the prop fingerprint model is further used to establish a prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vectors are obtained by the prop fingerprint model processing pre-acquired prop-effective video frames, and are used to represent the image features of the props in the prop-effective video frames; the first prop fingerprint vectors are combined with the second prop fingerprint vectors to determine whether the props in the target video frame are effective.

[0145] The storage medium in this embodiment can implement each process of the aforementioned model training method and achieve the same function and effect, which will not be repeated here.

[0146] Another embodiment of this specification provides a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the following process:

[0147] The target video frame for determining whether a prop is effective is obtained. The target video frame is then processed using the prop fingerprint model trained as described above to obtain the first prop fingerprint vector corresponding to the target video frame. The first prop fingerprint vector is used to represent the image features of the prop in the target video frame.

[0148] Obtain a pre-established prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vectors are obtained by the prop fingerprint model processing the pre-obtained prop activation video frames, and are used to represent the image features of the props in the prop activation video frames;

[0149] Based on the first prop fingerprint vector and the second prop fingerprint vector, determine whether the prop in the target video frame is effective.

[0150] Optionally, when the computer-executable instructions are executed by the processor, determining whether a prop in the target video frame is effective based on the first prop fingerprint vector and the second prop fingerprint vector includes: calculating the distance between the first prop fingerprint vector and each of the second prop fingerprint vectors; selecting a preset number of target second prop fingerprint vectors that are closest to the first prop fingerprint vector from among the second prop fingerprint vectors based on the distance; obtaining a first prop identifier of the prop in the target video frame; obtaining a target prop identifier corresponding to each of the target second prop fingerprint vectors; and determining whether a prop in the target video frame is effective based on the first prop identifier and the target prop identifier; wherein, the target prop identifier is the identifier of the prop in the video frame where the prop is effective, corresponding to the target second prop fingerprint vector.

[0151] Optionally, when the computer-executable instructions are executed by the processor, determining whether a prop in the target video frame is effective based on the first prop identifier and the target prop identifier includes: voting on each of the target prop identifiers, with the voting result indicating whether the corresponding target prop identifier is the same as the first prop identifier; if the proportion of preset voting results in the voting results exceeds a preset ratio, then the prop in the target video frame is determined to be effective; otherwise, the prop in the target video frame is determined to be ineffective; wherein, the preset voting result indicates that the corresponding target prop identifier is the same as the first prop identifier.

[0152] Optionally, when the computer-executable instructions are executed by the processor, acquiring a target video frame indicating whether the prop to be detected is effective includes: acquiring a target video indicating whether the prop to be detected is effective, and selecting the target video frame from the target video; after determining whether the prop in the target video frame is effective, the instructions further include: determining whether the prop in the target video is effective based on the determination result of the target video frame.

[0153] Optionally, when the computer-executable instruction is executed by the processor, it determines whether the props in the target video are effective based on the judgment result of the target video frame, including: if the judgment result of the target video frame indicates that the props in the target video frame are effective, then the props in the target video are determined to be effective; if the judgment result of the target video frame indicates that the props in the target video frame are not effective, then the props in the target video are determined to be not effective.

[0154] Optionally, when the computer-executable instructions are executed by the processor, the item fingerprint vector library is constructed in the following manner: for an item in the preset item library, obtain the item activation video frame corresponding to the item; input the item activation video frame into the item fingerprint model for processing, obtain the second item fingerprint vector corresponding to the item activation video frame output by the item fingerprint model; and construct the item fingerprint vector library using the second item fingerprint vector.

[0155] The storage medium in this embodiment can implement each process of the aforementioned item activation determination method and achieve the same function and effect, which will not be repeated here.

[0156] The computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc.

[0157] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0158] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0159] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0160] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0161] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0167] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0168] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A model training method, characterized in that, include: Construct training samples in the form of triples; the training samples include a baseline video frame, positive example video frames, and negative example video frames; The reference video frame and the positive example video frame have the same active props, and the background content of the reference video frame and the positive example video frame does not meet the preset similarity requirement; the reference video frame and the negative example video frame have different active props, and the background content of the reference video frame and the negative example video frame meets the preset similarity requirement. Using the training samples in the form of triples and a preset loss function, a pre-built neural network structure is trained; the loss function is used to ensure that the distance between the prop fingerprint vector corresponding to the benchmark video frame and the prop fingerprint vector corresponding to the positive example video frame is less than or equal to a first preset distance, and to ensure that the distance between the prop fingerprint vector corresponding to the benchmark video frame and the prop fingerprint vector corresponding to the negative example video frame is greater than or equal to a second preset distance; the second preset distance is greater than the first preset distance; The trained neural network structure is used as a prop fingerprint model; the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame; the determination result of whether the prop in the target video frame is effective is used to determine whether the prop in the target video is effective.

2. The method according to claim 1, characterized in that, Construct training samples in the form of triples, including: Obtain multiple second videos that do not use props, and use an automated script to add props to these second videos. From the multiple second videos after adding props, extract a reference video frame, a positive example video frame, and a negative example video frame; wherein, the reference video frame and the positive example video frame are second video frames with the same effective props but whose background content does not meet the preset similarity requirements; the reference video frame and the negative example video frame are second video frames with different effective props but whose background content meets the preset similarity requirements; Training samples in the form of triplets are constructed based on the extracted baseline video frames, positive example video frames, and negative example video frames.

3. The method according to any one of claims 1-2, characterized in that, The prop fingerprint model is also used to establish a prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vectors are obtained by the prop fingerprint model through processing pre-acquired prop-effective video frames, and are used to represent the image features of the props in the prop-effective video frames; the first prop fingerprint vector and the second prop fingerprint vector are combined to determine whether the props in the target video frame are effective.

4. A method for determining the effectiveness of an item, characterized in that, include: A target video frame is obtained to determine whether the prop is effective. The target video frame is then processed using the prop fingerprint model trained according to any one of claims 1-3 to obtain a first prop fingerprint vector corresponding to the target video frame. The first prop fingerprint vector is used to represent the image features of the prop in the target video frame. Obtain a pre-established prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vectors are obtained by the prop fingerprint model processing the pre-obtained prop activation video frames, and are used to represent the image features of the props in the prop activation video frames; Based on the first prop fingerprint vector and the second prop fingerprint vector, determine whether the prop in the target video frame is effective.

5. The method according to claim 4, characterized in that, Determining whether a prop in the target video frame is effective based on the first prop fingerprint vector and the second prop fingerprint vector includes: Calculate the distance between the first item fingerprint vector and each of the second item fingerprint vectors, and select a preset number of target second item fingerprint vectors that are closest to the first item fingerprint vector from among the second item fingerprint vectors based on the distance; Obtain the first item identifier of the item in the target video frame, obtain the target item identifier corresponding to each target second item fingerprint vector, and determine whether the item in the target video frame is effective based on the first item identifier and the target item identifier; wherein, the target item identifier is the identifier of the item in the video frame where the item is effective, corresponding to the target second item fingerprint vector.

6. The method according to claim 5, characterized in that, Determining whether a prop in the target video frame is effective based on the first prop identifier and the target prop identifier includes: Each of the target item identifiers is voted on separately, and the voting result is used to indicate whether the corresponding target item identifier is the same as the first item identifier; If the number of preset voting results in the voting results exceeds a preset proportion, then the prop in the target video frame is determined to be effective; otherwise, the prop in the target video frame is determined to be ineffective. The preset voting results are used to indicate that the corresponding target prop identifier is the same as the first prop identifier.

7. The method according to claim 4, characterized in that, Obtain the target video frame for determining whether the prop to be tested is effective, including: Obtain the target video to determine whether the prop to be tested is effective, and select the target video frame from the target video; After determining whether the props in the target video frame are effective, the method further includes: Based on the judgment result of the target video frame, determine whether the props in the target video are effective.

8. The method according to claim 7, characterized in that, Based on the judgment result of the target video frame, determine whether the props in the target video are effective, including: If the judgment result of the target video frame indicates that the prop in the target video frame is effective, then the prop in the target video is determined to be effective; if the judgment result of the target video frame indicates that the prop in the target video frame is not effective, then the prop in the target video is determined to be not effective.

9. The method according to claim 4, characterized in that, The prop fingerprint vector library is constructed in the following way: For each prop in the preset prop library, obtain the corresponding prop activation video frame; The video frame in which the prop takes effect is input into the prop fingerprint model for processing, and the second prop fingerprint vector corresponding to the video frame in which the prop takes effect is output by the prop fingerprint model is obtained. The prop fingerprint vector library is constructed using the second prop fingerprint vector.

10. A model training device, characterized in that, include: A sample construction unit is used to construct training samples in the form of triples; the training samples include a baseline video frame, positive example video frames, and negative example video frames. The reference video frame and the positive example video frame have the same active props, and the background content of the reference video frame and the positive example video frame does not meet the preset similarity requirement; the reference video frame and the negative example video frame have different active props, and the background content of the reference video frame and the negative example video frame meets the preset similarity requirement. The model training unit is used to train a pre-built neural network structure using training samples in the form of triples and a preset loss function; the loss function is used to ensure that the distance between the prop fingerprint vector corresponding to the benchmark video frame and the prop fingerprint vector corresponding to the positive example video frame is less than or equal to a first preset distance, and to ensure that the distance between the prop fingerprint vector corresponding to the benchmark video frame and the prop fingerprint vector corresponding to the negative example video frame is greater than or equal to a second preset distance; the second preset distance is greater than the first preset distance; The model generation unit is used to use the trained neural network structure as a prop fingerprint model; the prop fingerprint model is used to generate a first prop fingerprint vector corresponding to the target video frame to determine whether the prop in the target video frame is effective; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame; the determination result of whether the prop in the target video frame is effective is used to determine whether the prop in the target video is effective.

11. A prop activation determination device, characterized in that, include: The first acquisition unit is used to acquire a target video frame indicating whether the prop to be detected is effective, and to process the target video frame using the prop fingerprint model trained as described in claim 10 to obtain a first prop fingerprint vector corresponding to the target video frame; the first prop fingerprint vector is used to represent the image features of the prop in the target video frame. The second acquisition unit is used to acquire a pre-established prop fingerprint vector library; the prop fingerprint vector library stores multiple second prop fingerprint vectors; the second prop fingerprint vector is obtained by the prop fingerprint model processing the pre-acquired prop activation video frames, and is used to represent the image features of the props in the prop activation video frames; The effectiveness determination unit is used to determine whether the prop in the target video frame is effective based on the first prop fingerprint vector and the second prop fingerprint vector.

12. A model training device, characterized in that, include: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method described in any one of claims 1-3.

13. A device for determining the effectiveness of a prop, characterized in that, include: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method described in any one of claims 4-9.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer-executable instructions that, when executed by a processor, implement the steps of the method described in any one of claims 1-3.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer-executable instructions that, when executed by a processor, implement the steps of the method described in any one of claims 4-9.

Images