Video analysis method and device, electronic equipment, storage medium and product

Abstract

The invention relates to a video analysis method and device, electronic equipment, a storage medium and a product. The video analysis method comprises the following steps: acquiring a target video and a question corresponding to the target video, and generating an answer to the question based on a first video frame set in the target video; if a target answer corresponding to the question cannot be generated based on the first video frame set, a second video frame set is determined in the target video based on video frames in the first video frame set, and the second video frame set comprises video frames capable of being used for generating the target answer. Through the video analysis method provided by the embodiment of the invention, the stability of the video analysis process can be effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to video analysis methods, apparatus, electronic devices, storage media, and products. Background Technology

[0002] With the development of artificial intelligence (AI) technology, large language models (LLM) are widely used by users.

[0003] Taking the Multi-Large Language Model (M-LLM) as an example, users typically use the reasoning capabilities of the M-LLM to analyze questions and obtain the corresponding answers.

[0004] Among related technologies, video analysis based on AI models suffers from poor stability during the analysis process. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides a video analysis method, apparatus, electronic device, storage medium, and product.

[0006] According to a first aspect of some embodiments of this disclosure, a video analysis method is provided, including acquiring a target video and a question corresponding to the target video, and generating an answer to the question based on a first set of video frames in the target video; if a target answer corresponding to the question cannot be generated based on the first set of video frames, then determining a second set of video frames in the target video based on video frames in the first set of video frames, wherein the second set of video frames includes video frames that can be used to generate the target answer.

[0007] In one embodiment, determining a second set of video frames in the target video based on video frames in the first set of video frames includes: predicting a target video segment that is semantically related to the question based on the semantic association between video frames in the first set of video frames and the question; and determining the second set of video frames in the target video segment.

[0008] In one embodiment, predicting a target video segment semantically associated with the question based on the semantic association between video frames in the first video frame set and the question includes: determining, for a target video frame in the first video frame set, a semantic association order between a first semantic of the question and a second semantic of the target video frame, wherein the semantic association order includes the chronological order in which the semantics describe the content, and the target video frame is any video frame in the first video frame set; in response to the first semantic being associated with the second semantic first, determining the target segment in video segments before the corresponding frame number of the target video frame; and in response to the first semantic being associated with the second semantic later, determining the target segment in video segments after the corresponding frame number of the target video frame.

[0009] In one embodiment, determining the second video frame set in the target video segment includes: dividing the target video segment into multiple candidate video frame sets, and determining a first semantic association relationship between the first semantic of the problem and each candidate video frame set; and determining the candidate video frame set whose first semantic association relationship satisfies the semantic association relationship requirement as the second video frame set.

[0010] In one embodiment, the target answer is obtained by reasoning from the second set of video frames based on a pre-trained target model. The second set of video frames includes first video frames used in the historical reasoning process and second video frames used in the current reasoning process. The method further includes: performing image compression processing on the first video frames to obtain processed first video frames; and generating the target answer based on the second video frames, the processed first video frames, and the first semantics of the question.

[0011] According to a second aspect of some embodiments of this disclosure, a video analysis apparatus is provided, including an acquisition unit configured to acquire a target video and a question corresponding to the target video, and generate an answer to the question based on a first set of video frames in the target video; and a processing unit configured to, if the first set of video frames cannot generate a target answer corresponding to the question, determine a second set of video frames in the target video based on video frames in the first set of video frames, wherein the second set of video frames includes video frames that can be used to generate the target answer.

[0012] In one embodiment, the processing unit determines a second video frame set in the target video based on video frames in the first video frame set in the following manner: predicting a target video segment that is semantically related to the question based on the semantic association between the video frames in the first video frame set and the question; and determining the second video frame set in the target video segment.

[0013] In one implementation, based on the semantic association between video frames in the first video frame set and the question, the processing unit predicts target video segments semantically associated with the question in the following manner: for a target video frame in the first video frame set, determining the semantic association order between a first semantic of the question and a second semantic of the target video frame, wherein the semantic association order includes the chronological order in which the semantics describe the content, and the target video frame is any video frame in the first video frame set; in response to the first semantic being associated with the second semantic first, determining the target segment in video segments before the frame number corresponding to the target video frame; in response to the first semantic being associated with the second semantic later, determining the target segment in video segments after the frame number corresponding to the target video frame.

[0014] In one embodiment, in the target video segment, the processing unit determines the second video frame set in the following manner: dividing the target video segment into multiple candidate video frame sets, and determining the semantic association relationship between the first semantic of the question and the semantic of the video frame corresponding to each candidate video frame set; determining the candidate video frame set containing the video frame whose semantic association relationship meets the semantic association relationship requirements as the second video frame set.

[0015] In one embodiment, the target answer is obtained by reasoning from the second set of video frames based on a pre-trained target model. The second set of video frames includes first video frames used in the historical reasoning process and second video frames used in the current reasoning process. The processing unit is further configured to: perform image compression processing on the first video frames to obtain processed first video frames; and generate the target answer based on the second video frames, the processed first video frames, and the first semantics of the question.

[0016] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the video analysis method described in the first aspect or any embodiment of the first aspect.

[0017] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the video analysis method described in the first aspect or any embodiment of the first aspect.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, implements the video analysis method described in the first aspect or any embodiment of the first aspect.

[0019] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: When the target answer cannot be generated based on the first set of video frames, by determining the second set of video frames, a preliminary screening and determination of video frames that can be used to generate the target answer can be achieved. This increases the likelihood that the electronic device can determine video frames that can be used to generate the target answer based on the second set of video frames, thereby completing the analysis and reasoning of the video. It is evident that through the video analysis method proposed in the embodiments of this disclosure, the electronic device can adaptively determine video frames that can be used to generate the target answer in the target video based on the video analysis task. This weakens the coupling relationship between the electronic device and the first set of video frames during video analysis, thereby enhancing the stability and robustness of the electronic device in the face of long video analysis tasks.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] Figure 1A This is a flowchart illustrating a video analysis method according to some embodiments of the present disclosure.

[0023] Figure 1B This is a scene diagram illustrating a video analysis method according to some embodiments of the present disclosure.

[0024] Figure 2 This is a flowchart illustrating a method for determining a second set of video frames according to some embodiments of the present disclosure.

[0025] Figure 3 This is a flowchart illustrating a method for determining a target segment according to some embodiments of the present disclosure.

[0026] Figure 4 This is a flowchart illustrating a method for determining a second set of video frames according to some embodiments of the present disclosure.

[0027] Figure 5 This is a flowchart illustrating a method for generating a target answer according to some embodiments of the present disclosure.

[0028] Figure 6 This is a block diagram of a video analysis device according to some embodiments of the present disclosure.

[0029] Figure 7 This is a block diagram of an apparatus for video analysis according to some embodiments of the present disclosure.

[0030] Figure 8 This is a block diagram of an apparatus for video analysis illustrated according to some embodiments of the present disclosure. Figure 2 . Detailed Implementation

[0031] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0032] With the development of artificial intelligence (AI) technology, large language models (LLM) are widely used by users.

[0033] Taking the Multi-Large Language Model (M-LLM) as an example, in related technologies, users usually use the reasoning ability of M-LLM to analyze questions and obtain the corresponding answers.

[0034] For example, in some scenarios, video A might be a movie video, and a user might want to know some information presented in video A (e.g., what is the ending of character B in video A), but the user may not be able to watch video A themselves and confirm this information from it. In this case, the user can input video A into a pre-trained M-LLM (e.g., a model with video analysis capabilities) and input a corresponding question into the model (e.g., the question could be: what is the ending of character B in video A?). The model then performs reasoning analysis based on the question and the input video, and generates the target answer corresponding to the question (e.g., the generated answer could be: character B eventually became a pianist and achieved his dream).

[0035] Understandably, the target answer generated by M-LLM is determined based on the image in video A that is associated with the question. Continuing the example above, since the question includes the token "Character B," M-LLM may need to determine the answer to the question from images whose content includes "Character B."

[0036] However, continuing with the above example, in related technologies, M-LLM, upon acquiring an input video (e.g., video A), typically extracts a fixed number of image frames from the video for inference analysis to generate the target answer to the question. In this case, if the number of extracted image frames is small, it may be impossible to extract image frames suitable for answering the question (i.e., in the above example, it may be impossible to extract image frames including "role B"). If the number of extracted image frames is large, the model may face a large number of redundant image frames during inference, reducing inference efficiency and resulting in significant waste of computational resources.

[0037] In view of this, this disclosure proposes a video analysis method that, by setting a dynamic frame selection mechanism during video analysis, enables the model to re-determine the target image frames for generating the target answer from a second video frame set outside the first set when the model cannot generate the target answer based on an initial set of video frames (e.g., referred to as the first video frame set). Thus, the dynamic frame selection mechanism reduces the impact of the number of video frames used for inference during video analysis, reducing the possibility of inaccurate inference results or wasted computing resources due to an unreasonable number of video frames used for inference.

[0038] It should be noted that the video analysis method provided in this disclosure can be applied to electronic devices. Electronic devices may include, for example, terminals or servers. Terminals include, for example, mobile phones, wearable devices, IoT devices, automobiles with communication capabilities, smart cars, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminal devices in industrial control, wireless terminal devices in self-driving, wireless terminal devices in remote medical surgery, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, and wireless terminal devices in smart homes, but are not limited thereto. Servers may include, but are not limited to, independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0039] The embodiments disclosed herein can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0040] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0041] For ease of understanding, some technical terms involved in the embodiments of this disclosure will be explained by way of example below:

[0042] A video frame is the basic unit of a video. For example, it can be understood as the still image displayed on the screen at each pause during video playback. Furthermore, in some implementations, a video frame may include complete visual information at the corresponding moment (e.g., pixel color, brightness, etc.).

[0043] Semantic association relationship refers to the semantic connection between different semantics. These associations can be direct or indirect, and they help us understand the deeper meaning and structure between semantics. In this disclosure, semantic association relationships may include at least one of the following: synonymy, antonymy, hyponymy, hypernymy, meronymy, holonymy, causality, part-whole relationship, coordination, or subordination. In this disclosure, the target model can identify the semantics corresponding to the input question and the semantics of the video frames, and can determine the semantic association relationship between them based on their meanings. It then performs corresponding operations based on the determined semantic association relationship (e.g., selecting a second set of video frames or determining video frames that can be used to generate the target answer).

[0044] Semantic relatedness degree refers to a quantitative measure of the closeness and logical strength of the semantic connection between two linguistic units (such as words, phrases, sentences, and images). It goes beyond a simple binary judgment of "whether they are related" and instead describes the depth, proximity, and logical plausibility of the connection through numerical values ​​or levels. In this disclosure, the degree of semantic relatedness can be determined, for example, through assessment based on a human reference scale, rule calculation based on a knowledge base, or assessment based on a statistical learning model.

[0045] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0046] Figure 1A This is a flowchart illustrating a video analysis method according to some embodiments of the present disclosure, such as... Figure 1A As shown, the video analysis method is applied to electronic devices (e.g., terminals or servers) and includes the following steps S11 to S12.

[0047] In step S11, the target video and the question corresponding to the target video are obtained, and the answer to the question is generated based on the first set of video frames in the target video.

[0048] In step S12, if the target answer to the question cannot be generated based on the first set of video frames, then a second set of video frames is determined in the target video based on the video frames in the first set of video frames.

[0049] The second set of video frames includes video frames that can be used to generate the target answer.

[0050] In this embodiment, since the electronic device cannot determine the target answer based on the first set of video frames, it can be assumed that the first set of video frames does not include any video frames that can be used to generate the target answer. Furthermore, since the second set of video frames includes video frames that can be used to generate the target answer, when the target answer cannot be generated based on the first set of video frames, determining the second set of video frames allows for a preliminary screening and determination of video frames that can be used to generate the target answer. This increases the likelihood that the electronic device can determine video frames that can be used to generate the target answer based on the second set of video frames, thereby completing the video analysis and reasoning. Therefore, through the video analysis method proposed in this embodiment, the electronic device can adaptively determine video frames that can be used to generate the target answer in the target video based on the video analysis task. This weakens the coupling between the electronic device and the first set of video frames during video analysis, thereby enhancing the stability and robustness of the electronic device in the face of long video analysis tasks.

[0051] It is understood that when video analysis methods are applied to electronic devices, for example, when video analysis methods are applied to electronic devices that have deployed a target model (e.g., a pre-trained multimodal large language model with video analysis capabilities), the electronic devices execute the relevant steps of the above implementation method by calling the target model.

[0052] To facilitate understanding, the process of performing steps S11 to S12 on an electronic device with the target model deployed will be explained below with reference to the following examples A1) and B1).

[0053] A1) In some implementations, step S11 can be understood as the electronic device inputting the acquired target video and the question corresponding to the target video into the target model, and calling the target model to perform the task of "generating an answer" based on the first set of video frames in the target video to achieve video analysis.

[0054] B1) In some implementations, step S12 can be understood, for example, as follows: if the target model cannot reason about the target answer based on the first video set, then the target model determines the second video set in the target video based on the video frames in the first video set.

[0055] To facilitate understanding, the following will be explained... Figure 1B The scenarios shown illustrate the application scenarios corresponding to examples A1) and B1) above. Figure 1B This is a scene diagram illustrating a video analysis method according to some embodiments of the present disclosure.

[0056] like Figure 1B As shown, the target model in the figure can be understood, for example, as a pre-trained multimodal large language model with video analysis capabilities, deployed on an electronic device. The model's input, as shown, includes two parts: the target video and the question. The target video can be understood, for example, as a recorded video about character A. The question could be, for example,: What is the relationship between character A and character B?

[0057] The first video frame set can be understood as a collection of video frames from the target device. This first video frame set can be a pre-defined set (e.g., the first video frame set can be input simultaneously during the input of the target video to the target model) used by the target model for initial inference. Alternatively, the first video frame set can be a set of video frames extracted by the target model from the target video based on pre-set rules (e.g., random extraction, or extraction based on a fixed period).

[0058] Understandably, in order to generate the target answer to the corresponding question, the target model needs to identify the video frames associated with "role A" (e.g., video frames in the image content that include role A) and the video frames associated with "role B" (e.g., video frames in the image content that include role B).

[0059] It is also understandable that the first set of video frames may not include video frames that can be used to generate the target answer (e.g., video frames associated with "Character A" and / or video frames associated with "Character B"). In this case, the target model cannot generate the target answer corresponding to the question based on the first set of video frames. Therefore, the target model needs to determine, based on the first true set of video frames, which in the target video include video frames that can be used to generate the target answer (e.g., video frames associated with "Character A" and / or video frames associated with "Character B").

[0060] To determine the video frames used to generate the target answer for the target model, we can first estimate the approximate range corresponding to the video frame, or the relative position of the set of video frames it belongs to (i.e., the second set of video frames) within the target video (e.g., the set of video frames corresponding to a certain frame number range). Then, we use the second set of video frames to determine the video frames that can be used to generate the target answer, and then generate the answer based on the video frames (e.g., the answer in the figure: Character A and Character B are partners because they are smiling and walking together), thus completing the video analysis.

[0061] Therefore, in the above implementation scenario, the electronic device analyzes the target video by calling the target model.

[0062] Based on the above description of the relevant implementation methods, it can be understood that not all video frames in the target video can be used as video frames for generating the target answer. That is, the video frames that can be used to generate the target answer may have features related to the question in a certain dimension.

[0063] For example, in some implementations, such as those involving a target video, if the semantics corresponding to a video frame have a semantic association with the semantics corresponding to the question, then that video frame can be used as a video frame capable of generating the target answer. Thus, in some implementations, a second set of video frames can be determined based on the semantic association between the set of video frames and the question.

[0064] Figure 2 This is a flowchart illustrating a method for determining a second set of video frames according to some embodiments of this disclosure. Figure 2 As shown, the method includes the following steps S21 to S22.

[0065] In step S21, based on the semantic association between the video frames in the first video frame set and the question, the target video segment that has a semantic association with the question is predicted.

[0066] In step S22, a second set of video frames is determined within the target video segment.

[0067] In this embodiment, by determining the semantic relationship between video frames in the first video frame set and the question, target video segments semantically related to the question are identified, thus achieving preliminary screening of the second video frame set. This allows for a progressive determination of the second video frame set based on the target segments. On one hand, this significantly improves the density of effective information (e.g., information about the second video frame set) compared to a full-input approach in determining the second video frame set. On the other hand, the progressive determination of the second video frame set, compared to a full-input approach, reduces computational complexity and improves task processing efficiency.

[0068] It should be noted that a video segment can be understood as a video fragment comprising one or more sets of video frames (in some scenarios, it can also be the complete target video). A target video segment can be immediately defined as a video segment comprising a second set of video frames (the target video segment may also include other sets of video frames besides the second set). A video segment can be obtained, for example, by dividing the target video.

[0069] For ease of understanding, the following will continue from the above. Figure 1B Mid-scene Figure 2 The implementation methods are illustrated by way of example.

[0070] like Figure 1B As shown, assume that the first set of video frames includes video frame 1, video frame 2, and video frame 3. The semantics corresponding to video frame 1 could be, for example, "seaside scenery," the semantics corresponding to video frame 2 could be, for example, "a photo of character A as an infant," and the semantics corresponding to video frame 3 could be, for example, "a car." And assume the semantics of the question is "What is the relationship between character A and character B?"

[0071] Understandably, the semantic relationship between the semantics of video frame 1 and the semantics corresponding to the question (hereinafter referred to as the first semantic relationship for simplicity) is relatively weak. The semantic relationship between the semantics of video frame 2 and the semantics corresponding to the question (hereinafter referred to as the second semantic relationship for simplicity) is stronger than the first semantic relationship. The semantic relationship between the semantics of video frame 3 and the semantics corresponding to the question (hereinafter referred to as the third semantic relationship for simplicity) is weaker than the second semantic relationship.

[0072] It is evident that the second semantic association is stronger than the first semantic association, and the second semantic association is stronger than the third semantic association. Therefore, the target video segment can be predicted based on video frame 2 (e.g., the target video segment is predicted to be a video segment that comes before video frame 2, or the target video segment is predicted to be a video segment that comes after video frame 2, etc.).

[0073] Therefore, the second video frame set can be determined through the above implementation method.

[0074] The target video frequency band can be determined, for example, by the following methods: Figure 3 The implementation method shown is adopted. Figure 3 This is a flowchart illustrating a method for determining a target segment according to some embodiments of this disclosure. Figure 3As shown, the method includes the following steps S31 to S32-2.

[0075] In step S31, for the target video frame in the first video frame set, the semantic association order between the first semantic of the question and the second semantic of the target video frame is determined.

[0076] In step S32-1, in response to the first semantic being associated with the second semantic prior to it, the target segment is determined in the video segment preceding the frame number corresponding to the target video frame.

[0077] In step S32-2, in response to the first semantic being associated with the second semantic, the target segment is determined in the video segment after the frame number corresponding to the target video frame.

[0078] The semantic association order includes the chronological order in which the semantically described content occurs, and the target video frame is any video frame in the first set of video frames.

[0079] In this embodiment of the disclosure, since the semantic association order includes the chronological order of the content described by the semantics, the chronological relationship between the target video segment and the target video frame can be determined by the semantic association order between the first semantic and the second semantic, and then the target video segment can be determined in the target video based on the target video frame.

[0080] For ease of understanding, the following will continue from the above. Figure 2 The example scenarios in the middle Figure 3 Exemplary implementations are described below.

[0081] Taking the implementation of step S32-1 above as an example: (Continued) Figure 1B The description of the relevant implementation method assumes that "Character B" is unrelated to "Character A" when "Character A is an infant" (for example, "Character A" is an infant and does not know "Character B"). It is understood that the semantics (i.e., the second semantics) in video frame 2 includes a description of "Character A" but not of "Character B." Based on the foregoing, it is known that "Character B" is highly likely to appear in video segments after video frame 2 (i.e., video segments corresponding to the period after "Character A" is an infant). In other words, the first semantics is the semantics associated with the second semantics. Therefore, the target video segment is predicted to be a video segment after video frame 2, and the video frame set can be filtered within the video segment after video frame 2 to determine the second video frame set.

[0082] It should be noted that the implementation of step S32-2 above can also be confirmed by referring to the methods mentioned in the above examples. For ease of understanding, they will not be described in detail here.

[0083] Therefore, through the above-described related implementation methods, the target video segment can be determined, and the second video frame set can be removed from the target video segment, for example, through the following... Figure 4 The implementation method shown is adopted.

[0084] Figure 4 This is a flowchart illustrating a method for determining a second set of video frames according to some embodiments of this disclosure. Figure 4 As shown, the method includes the following steps S41 to S42.

[0085] In step S41, the target video segment is divided into multiple candidate video frame sets, and the first semantic of the problem and the semantic association between the video frames in each candidate video frame set are determined.

[0086] In step S42, the set of candidate video frames containing video frames whose semantic association meets the semantic association requirements is determined as the second video frame set.

[0087] In this embodiment of the disclosure, since the video segment includes one or more sets of video frames, the target video segment is divided into multiple sets of candidate video frames, and the set of candidate video frames related to the problem is determined based on the semantics of each video frame in the set of candidate video frames. Then, a second set of video frames is determined from the set of candidate video frames related to the problem.

[0088] It should be noted that for semantic association to meet the semantic association requirements, for example, it can be understood that the degree of association of the semantic association meets the degree of association requirements (for the distinction between the strength of semantic association, please refer to the relevant implementation methods mentioned above, which will not be repeated here).

[0089] To facilitate understanding, the following will combine the above... Figure 1B The relevant implementation scenarios and the aforementioned implementation methods, for Figure 4 The relevant implementation methods are described by way of example.

[0090] Following the aforementioned related implementation methods, the target video segment can be divided into candidate video frame set 1, candidate video frame set 2, and candidate video frame set 3, for example.

[0091] The candidate video frame set 1 includes, for example, video frame 4 and video frame 5. The semantic meaning corresponding to video frame 4 could be, for example, "pet dog," and the semantic meaning corresponding to video frame 5 could be, for example, "house."

[0092] The candidate video frame set 2 includes, for example, video frame 6, video frame 7 and video frame 8. The semantics corresponding to video frame 6 could be, for example, "school", the semantics corresponding to video frame 7 could be, for example, "character B", and the semantics corresponding to video frame 8 could be, for example, "character A and character B are taking a walk at school".

[0093] The candidate video frame set 3 may include, for example, video frame 9 and video frame 10. The semantics corresponding to video frame 9 may be, for example, "character A's graduation photo", and the semantics corresponding to video frame 10 may be, for example, "graduation ball".

[0094] It is understandable that if the semantic relationship between the semantics corresponding to video frame 8 and the first semantics satisfies the correlation relationship (e.g., the degree of correlation meets the correlation requirement), then the candidate video frame set containing video frame 8 (i.e., candidate video frame set 2) can be determined as the second video frame set.

[0095] Therefore, the second video set can be determined through the above implementation method.

[0096] In summary, electronic devices can determine the second video set by calling the target model, and then determine the video frames that can be used to generate the target answer, thereby completing video analysis.

[0097] For example, in some implementations, the target model is based on a second set of videos, which determines the video frames that can be used to generate the answer, and inferences are performed based on the video frames to generate the target answer.

[0098] Understandably, in some implementation scenarios, the target model performs inference based on videos that can be used to generate answers. This inference process may involve multiple rounds, and each round may involve determining the video frames used to generate the answer multiple times. Therefore, to reduce the computational overhead of the target model during inference and ensure its stable operation even with complex inference tasks, the images used in historical inference processes can be compressed. This reduces the target model's memory usage and lowers the computational cost of the inference process.

[0099] For example, Figure 5 This is a flowchart illustrating a method for generating a target answer according to some embodiments of this disclosure. Figure 5 As shown, the method includes the following steps S51 to S52.

[0100] In step S51, the first video frame is subjected to image compression processing to obtain the processed first video frame.

[0101] In step S52, the target answer is generated based on the second video frame, the processed first video frame, and the first semantics of the question.

[0102] The target answer is obtained by reasoning from the second set of video frames based on the pre-trained target model. The second set of video frames includes the first video frames used in the historical reasoning process and the second video frames used in the current reasoning process.

[0103] In this embodiment, on the one hand, by compressing the first video frame used in the historical reasoning process, its memory usage on the model can be reduced. On the other hand, by combining the compressed first video frame with the second video frame used in the current reasoning process to execute the current reasoning process, the reasoning process can be associated with the context, making the reasoning result more accurate.

[0104] Therefore, the above-mentioned implementation methods can be used to generate the target answer, thereby enabling the analysis and reasoning of the target video.

[0105] In some implementations, the target model can be obtained, for example, by performing the following process A2) to B2) on a pre-trained base model (e.g., a base large language model).

[0106] A2) Supervision and fine-tuning.

[0107] For example, the main parameters of the visual encoder and the base model can be frozen, and only the projection layer and the functional modules used to specify the functions (e.g., the functional modules used to perform the related implementations described above) can be fine-tuned.

[0108] Among these, training objectives can guide the model to dynamically select key segments based on the current inference context and compress historical context information, while also learning to determine when inference terminates.

[0109] The training dataset used in the supervised fine-tuning process may include at least one of the following: training videos, questions, start-of-reasoning markers, inference step instructions, end-of-reasoning markers, and answers.

[0110] The loss function can be designed as follows (1):

[0111]

[0112] Where θ represents the model parameters to be optimized (e.g., the parameters of the projection layer and the aforementioned functional modules), V represents the input video, x represents the question text, and r represents the input video. i y represents the video segment (i.e., the set of video frames) dynamically selected in the i-th step. i Represents the reasoning text generated in step i, {r i ,y i ,o j} represents the segment selection, text output, and visual features of the historical steps, and π represents the generation of r. i ,y i The conditional probability.

[0113] Therefore, by designing the above loss function, it is possible to simultaneously optimize the joint probability of dynamic video frame combination selection and text inference generation.

[0114] B2) Reinforcement learning phase.

[0115] After the model undergoes the A2) processing described above, reinforcement learning training can be performed using Group Relative Policy Optimization (GRPO). For each input question x and video V, the model samples G inference processes Y1, Y2, ..., Y from the current model. G Each reasoning process calculates a reward based on the correctness of the reasoning and the length of the reasoning process, and generates an advantage function A based on the relative ranking within the group. i,t The optimization objective is to maximize the expectation expressed by the following expression (2):

[0116]

[0117] Where G represents the number of inference paths in parallel sampling (i.e., the group size). Y i Let i represent the i-th reasoning path (including multi-step decision sequences). p represents the indicator function (e.g., 1 when the t-th step of the i-th inference path is valid). t A represents the ratio of the probabilities of the new and old strategies (used to control the magnitude of strategy updates). i,t denoted as the dominance function, and ∈ denotes the shearing range hyperparameter (e.g., a typical value can be 0.1 to 0.2).

[0118] Therefore, by optimizing the reasoning path through group-based relative strategies, computational complexity can be reduced while avoiding the problem of reward sparsity.

[0119] Therefore, the above-mentioned implementation methods can achieve the determination of the target model. Firstly, the target model enables dynamic adjustment of the focus window during inference, effectively capturing fine-grained key information in long videos and overcoming the limitation of traditional static frame selection methods that cannot simultaneously consider both the overall picture and details. Secondly, the target model can dynamically compress historical visual information based on the progress of inference, retaining more important video content within a limited context space, improving inference coherence and information utilization. Thirdly, the target model can dynamically update the observation window and rationally manage context information, effectively avoiding breaks or information loss during inference, enhancing the model's stability and robustness in complex inference tasks. Fourthly, by using the target model for video analysis, the video analysis method becomes applicable to various long video understanding scenarios such as long video question answering, event detection, behavior analysis, and security monitoring, possessing good versatility and practical application value.

[0120] Figure 6 This is a block diagram of a video analysis device according to some embodiments of the present disclosure. (Refer to...) Figure 6 The device 100 includes an acquisition unit 101 and a processing unit 102.

[0121] The acquisition unit 101 is used to acquire the target video and the question corresponding to the target video, and generate an answer to the question based on the first set of video frames in the target video.

[0122] The processing unit 102 is configured to determine a second set of video frames in the target video based on the video frames in the first set of video frames if the target answer to the question cannot be generated based on the first set of video frames. The second set of video frames includes video frames that can be used to generate the target answer.

[0123] In one embodiment, the processing unit 102 determines a second set of video frames in the target video based on video frames in a first set of video frames in the following manner: based on the semantic association between video frames in the first set of video frames and the question, it predicts a target video segment that is semantically related to the question; and in the target video segment, it determines a second set of video frames.

[0124] In one embodiment, based on the semantic association between video frames in the first video frame set and the question, the processing unit 102 predicts the target video segment that is semantically associated with the question in the following manner: for the target video frame in the first video frame set, the semantic association order between the first semantic of the question and the second semantic of the target video frame is determined, wherein the semantic association order includes the chronological order of the content described by the semantics, and the target video frame is any video frame in the first video frame set; in response to the first semantic being associated with the second semantic first, the target segment is determined in the video segment before the corresponding frame number of the target video frame; in response to the first semantic being associated with the second semantic later, the target segment is determined in the video segment after the corresponding frame number of the target video frame.

[0125] In one embodiment, in the target video segment, the processing unit 102 determines the second video frame set in the following manner: the target video segment is divided into multiple candidate video frame sets, and the semantic association relationship between the first semantic of the question and the semantic of the video frame in each candidate video frame set is determined; the candidate video frame set containing the video frame whose semantic association relationship meets the semantic association relationship requirement is determined as the second video frame set.

[0126] In one implementation, the target answer is obtained by reasoning from a second set of video frames based on a pre-trained target model. The second set of video frames includes the first video frames used in the historical reasoning process and the second video frames used in the current reasoning process. The processing unit 102 is further configured to: perform image compression processing on the first video frames to obtain the processed first video frames; and generate the target answer based on the second video frames, the processed first video frames, and the first semantics of the question.

[0127] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0128] Figure 7 This is a block diagram of an apparatus for video analytics according to some embodiments of the present disclosure. For example, apparatus 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0129] Reference Figure 7 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0130] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0131] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0132] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0133] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0134] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0135] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0136] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0137] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 216 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0138] In some embodiments of this disclosure, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0139] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by the processor 220 of the device 200 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0140] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0141] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of the device 200, the device 200 is able to perform the methods described above.

[0142] Figure 8 This is a block diagram of an apparatus for video analysis illustrated according to some embodiments of the present disclosure. Figure 2 For example, device 300 can be provided as a server. (See reference...) Figure 8 The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the methods described above.

[0143] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0144] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 332 including instructions, which can be executed by the processing component 322 of the device 300 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0145] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0146] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of device 300, device 300 is able to perform the methods described above.

[0147] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this disclosure can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented in hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this disclosure.

[0148] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, a first component, part, region, layer, or section mentioned in the examples may also be referred to as a second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature.

[0149] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0150] In this description, "multiple" means at least two, referring to two or more, such as two, three, etc., unless otherwise explicitly specified. Other quantifiers are similar. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, unless otherwise specified or clearly indicated from the context, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more."

[0151] In this description, the terms “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “if…”, “if…”, etc. can be used interchangeably.

[0152] In this description, the terms “greater than”, “greater than or equal to”, “not less than”, “more than”, “more than or equal to”, “not less than”, “higher than”, “higher than or equal to”, “not lower than”, and “above” can be used interchangeably. The terms “less than”, “less than or equal to”, “not greater than”, “less than”, “less than or equal to”, “not more than”, “lower than”, “lower than or equal to”, “not higher than”, and “below” can be used interchangeably.

[0153] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term "and / or" includes any one of the related listed items and any combination of two or more; "and / or" describes the association relationship between related objects, indicating that three relationships may exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Similarly, "at least one of..." includes any one of the related listed items and any combination of two or more.

[0154] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0155] Furthermore, the term "exemplary" is used herein to indicate that it serves as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term "exemplary" is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to indicate an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to indicate any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then applying A or B satisfies the condition under any of the foregoing instances.

[0156] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if it is not structurally equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in this disclosure, such terms are intended to be inclusive in a manner similar to the term “including.”

[0157] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0158] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video analysis method, characterized in that, include: Obtain the target video and the question corresponding to the target video, and generate an answer to the question based on the first set of video frames in the target video; If the target answer to the question cannot be generated based on the first set of video frames, then a second set of video frames is determined in the target video based on the video frames in the first set of video frames, wherein the second set of video frames includes video frames that can be used to generate the target answer.

2. The method according to claim 1, characterized in that, Determining a second set of video frames in the target video based on video frames in the first set of video frames includes: Based on the semantic association between video frames in the first video frame set and the question, predict the target video segment that has a semantic association with the question; Within the target video segment, the second set of video frames is determined.

3. The method according to claim 2, characterized in that, The step of predicting target video segments semantically related to the question based on the semantic association between video frames in the first video frame set and the question includes: For a target video frame in the first video frame set, determine the semantic association order between the first semantic of the question and the second semantic of the target video frame, wherein the semantic association order includes the chronological order in which the content described by the semantics occurs, and the target video frame is any video frame in the first video frame set; In response to the first semantic being a semantic associated with the second semantic prior to it, the target segment is determined in the video segment preceding the frame number corresponding to the target video frame; In response to the first semantic being a semantic associated with the second semantic, the target segment is determined in the video segment following the frame number corresponding to the target video frame.

4. The method according to claim 2, characterized in that, In the target video segment, the second set of video frames is determined, including: The target video segment is divided into multiple candidate video frame sets, and the semantic association between the first semantic of the question and the semantics corresponding to the video frames in each candidate video frame set is determined. The set of candidate video frames containing video frames whose semantic association meets the semantic association requirements is determined as the second set of video frames.

5. The method according to claim 1, characterized in that, The target answer is obtained by reasoning the second set of video frames based on the pre-trained target model. The second set of video frames includes the first video frames used in the historical reasoning process and the second video frames used in the current reasoning process. The method further includes: The first video frame is subjected to image compression processing to obtain the processed first video frame; The target answer is generated based on the second video frame, the processed first video frame, and the first semantics of the question.

6. A video analysis device, characterized in that, include: An acquisition unit is used to acquire a target video and a question corresponding to the target video, and to generate an answer to the question based on a first set of video frames in the target video; The processing unit is configured to determine a second video frame set in the target video based on the video frames in the first video frame set if the target answer to the question cannot be generated from the first video frame set, wherein the second video frame set includes video frames that can be used to generate the target answer.

7. The apparatus according to claim 6, characterized in that, The processing unit determines the second video frame set in the target video based on the video frames in the first video frame set in the following manner: Based on the semantic association between video frames in the first video frame set and the question, predict the target video segment that has a semantic association with the question; Within the target video segment, the second set of video frames is determined.

8. The apparatus according to claim 7, characterized in that, Based on the semantic association between video frames in the first video frame set and the question, the processing unit predicts target video segments that are semantically related to the question in the following manner: For a target video frame in the first video frame set, determine the semantic association order between the first semantic of the question and the second semantic of the target video frame, wherein the semantic association order includes the chronological order in which the content described by the semantics occurs, and the target video frame is any video frame in the first video frame set; In response to the first semantic being a semantic associated with the second semantic prior to it, the target segment is determined in the video segment preceding the frame number corresponding to the target video frame; In response to the first semantic being a semantic associated with the second semantic, the target segment is determined in the video segment following the frame number corresponding to the target video frame.

9. The apparatus according to claim 7, characterized in that, In the target video segment, the processing unit determines the second set of video frames in the following manner: The target video segment is divided into multiple candidate video frame sets, and the semantic association between the first semantic of the question and the semantics corresponding to the video frames in each candidate video frame set is determined. The set of candidate video frames containing video frames whose semantic association meets the semantic association requirements is determined as the second set of video frames.

10. The apparatus according to claim 6, characterized in that, The target answer is obtained by reasoning the second set of video frames based on the pre-trained target model. The second set of video frames includes the first video frames used in the historical reasoning process and the second video frames used in the current reasoning process. The processing unit is also used for: The first video frame is subjected to image compression processing to obtain the processed first video frame; The target answer is generated based on the second video frame, the processed first video frame, and the first semantics of the question.

11. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions that can be executed by a processor; The processor is configured to execute the computer program or instructions to implement the steps of the video analysis method according to any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by a processor of an electronic device, enable the processor of the electronic device to perform the video analysis method according to any one of claims 1 to 5.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the video analysis method as described in any one of claims 1 to 5.

Images