Information processing apparatus, information processing method, and computer-readable storage medium

Abstract

The application discloses an information processing device and method based on zero-order learning and a computer readable storage medium. The information processing device comprises: an activation heat map generation unit configured to generate an activation heat map of each frame in a video based on a predetermined request by using a pre-trained multi-modal model or a pre-trained attention network; a spatial region of interest determination unit configured to determine a region of interest of each frame based on the activation heat map of the frame; and a similarity time sequence obtaining unit configured to calculate a similarity between the region of interest of each frame and the predetermined request by using the pre-trained multi-modal model to obtain a similarity time sequence of the video, which can be used to identify a target frame corresponding to the predetermined request in the video.

Description

Technical Field

[0001] This disclosure relates to the field of information processing, and specifically to information processing apparatus, information processing methods, and computer-readable storage media. Background Technology

[0002] Users typically want to watch video segments that they specifically desire. Therefore, it is desirable to be able to identify the frames (i.e., images) that the user desires from the video. Summary of the Invention

[0003] A brief overview of this disclosure is given below to provide a basic understanding of certain aspects of it. However, it should be understood that this overview is not an exhaustive summary of this disclosure. It is not intended to identify key or essential parts of this disclosure, nor is it intended to limit the scope of this disclosure. Its purpose is merely to present certain concepts of this disclosure in a simplified form as a prelude to the more detailed description that follows.

[0004] In view of the above problems, the object of this disclosure is to provide an information processing apparatus, information processing method and computer-readable storage medium that enables accurate extraction of frames (i.e., images) corresponding to a predetermined request from a video.

[0005] According to one aspect of this disclosure, an information processing apparatus based on zero-learning is provided, comprising: an activation heatmap generation unit configured to generate an activation heatmap of each frame in a video based on a predetermined request using a pre-trained multimodal model or a pre-trained attention network; a spatial region of interest determination unit configured to determine a region of interest of each frame based on the activation heatmap of the frame; and a similarity time series acquisition unit configured to calculate the similarity between the region of interest of each frame and the predetermined request using the pre-trained multimodal model to obtain a similarity time series of the video, wherein the similarity time series can be used to identify a target frame in the video corresponding to the predetermined request.

[0006] According to another aspect of this disclosure, an information processing method based on zero-learning is provided, comprising: an activation heatmap generation step, for each frame in a video, generating an activation heatmap of the frame based on a predetermined request using a pre-trained multimodal model or a pre-trained attention network; a spatial region of interest determination step, for each frame, determining a region of interest of the frame based on the activation heatmap of the frame; and a similarity time series acquisition step, for calculating the similarity between the region of interest of each frame and the predetermined request using a pre-trained multimodal model to obtain a similarity time series of the video, wherein the similarity time series can be used to identify a target frame in the video corresponding to the predetermined request.

[0007] In accordance with other aspects of this disclosure, computer program code and computer program products for implementing the methods according to this disclosure are also provided, as well as a computer-readable storage medium having the computer program code for implementing the methods according to this disclosure recorded thereon.

[0008] Other aspects of embodiments of this disclosure are set forth in the following description section, wherein preferred embodiments of the present disclosure are described in detail without limiting them. Attached Figure Description

[0009] This disclosure can be better understood by referring to the detailed description given below in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar parts. These drawings, together with the following detailed description, are incorporated in and form part of this specification, and are used to further illustrate preferred embodiments of the disclosure and explain the principles and advantages of the disclosure. Wherein:

[0010] Figure 1 This is a block diagram illustrating an example of the functional configuration of an information processing apparatus based on zero-learning according to an embodiment of the present disclosure;

[0011] Figure 2 This is a schematic diagram illustrating an example architecture of an information processing apparatus based on zero-learning according to an embodiment of the present disclosure;

[0012] Figure 3 A schematic diagram showing an example frame and the region of interest determined for that example frame;

[0013] Figure 4A and Figure 4B Examples of a method for directly calculating the similarity between a frame and a predetermined request (hereinafter referred to as the "first method") and an example of a similarity time series obtained by an information processing apparatus according to an embodiment of the present disclosure are shown respectively.

[0014] Figure 5Examples of target frame identification results are shown using similarity time series obtained by the first method and the information processing apparatus 100 according to an embodiment of the present disclosure, respectively.

[0015] Figure 6 An example is shown of the similarity between a frame and a predetermined request calculated using the first method and the information processing device 100;

[0016] Figure 7 This is a block diagram illustrating an example of the functional configuration of an information processing apparatus based on zero-learning according to another embodiment of the present disclosure;

[0017] Figure 8 This is a flowchart illustrating an example of a zero-learning-based information processing method according to an embodiment of the present disclosure;

[0018] Figure 9 This is a flowchart illustrating an example of a zero-learning-based information processing method according to another embodiment of the present disclosure; and

[0019] Figure 10 This is a block diagram illustrating an example structure of a personal computer that may be employed in embodiments of this disclosure. Detailed Implementation

[0020] Exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual implementations are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the development of any such actual embodiment to achieve the developer’s specific goals, such as complying with constraints related to the system and business, and these constraints may vary from implementation to implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from the present disclosure.

[0021] It should also be noted that, in order to avoid obscuring this disclosure with unnecessary details, only the equipment structure and / or processing steps closely related to the solution according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0022] The embodiments according to this disclosure are described in detail below with reference to the accompanying drawings.

[0023] First, refer to Figure 1 , Figure 2 and Figure 3 An implementation example of a zero-learning-based information processing apparatus 100 according to an embodiment of the present disclosure is described. Figure 1This is a block diagram illustrating an example of the functional configuration of an information processing apparatus 100 based on zero-learning according to an embodiment of the present disclosure. Figure 2 This is a schematic diagram illustrating an example architecture of an information processing apparatus 100 based on zero-learning according to an embodiment of the present disclosure. Figure 3 A schematic diagram showing an example frame and the region of interest determined for that example frame.

[0024] like Figure 1 and Figure 2 As shown, the zero-learning-based information processing apparatus 100 according to an embodiment of the present disclosure may include an activation heatmap generation unit 102, a spatial region of interest determination unit 104, and a similarity time series acquisition unit 106.

[0025] The activation heatmap generation unit 102 can be configured to generate an activation heatmap for each frame in the video, based on a predetermined request, using a pre-trained multimodal model or a pre-trained attention network. For example, the value of each pixel in the activation heatmap (i.e., the value of each point) represents the degree of association between each pixel in the frame corresponding to the activation heatmap and the predetermined request.

[0026] As an example, the activation heatmap generation unit 102 can, for each frame, extract the features of a predetermined request and the features of the frame using a pre-trained multimodal model, use the features of the predetermined request as the output of a predetermined deep neural network model and the features of the frame as the input of the predetermined deep neural network model, and generate the activation heatmap of the frame using an interpretable method (such as Grad-CAM).

[0027] As another example, the activation heatmap generation unit 102 can, for each frame, use the attention module of a pre-trained multimodal model or a pre-trained attention network to calculate the attention weights of that frame based on the features requested in a predetermined manner, thereby generating the activation heatmap of that frame.

[0028] As an example, a reservation request may include at least one of text, images, and voice, but is not limited to these.

[0029] For example, a reservation request can involve a reserved object, such as a person, other animals, or an object. Furthermore, a reservation request can involve a reserved event, such as a person's action (e.g., raising a hand, dancing), a person's interaction with other objects (e.g., a person sitting in a chair, a person holding a box, a person eating an apple), or interactions between other objects (e.g., a dog fetching a frisbee), and so on. Moreover, a reservation request can also involve a reserved state or abstract description, such as "he looks very hesitant" or "a romantic scene."

[0030] As an example, the pre-trained multimodal model could be a pre-trained text-visual model. In this case, when the pre-defined request is presented in a form other than text, for example, the activation heatmap generation unit 102 can convert the pre-defined request into text form, and then use the pre-trained text-visual model to generate an activation heatmap based on the converted text form of the pre-defined request. Of course, those skilled in the art can use other pre-trained multimodal models, such as Image-GPT, as needed.

[0031] The spatial region of interest determination unit 104 can be configured to determine the region of interest for each frame based on the frame's activation heatmap. For example, the region of interest for a frame could be a region in the frame that has a high similarity to a predetermined request.

[0032] For example, Figure 3 This diagram illustrates the region of interest (highlighted with a dashed box) determined by the spatial region of interest determination unit 104 in the case of a predetermined request for the event "the person holding the box". Figure 3 It is evident that the spatial region of interest determination unit 104 can effectively determine the region of interest corresponding to the predetermined request.

[0033] The similarity time series acquisition unit 106 can be configured to use a pre-trained multimodal model to calculate the similarity between the region of interest of each frame and a predetermined request, thereby obtaining a similarity time series of the video. This similarity time series can be used to identify the target frame in the video corresponding to the predetermined request. The pre-trained multimodal model used by the similarity time series acquisition unit 106 can be the same as or different from the pre-trained multimodal model used by the activation heatmap generation unit 102.

[0034] For example, the similarity time series acquisition unit 106 can use a pre-trained multimodal model to extract features of the region of interest (ROI) and the features of the predetermined request for each frame, and calculate the similarity between the ROI and the predetermined request for each frame based on the extracted features. For instance, the similarity time series acquisition unit 106 can calculate the similarity between the ROI and the predetermined request for each frame in such a way that the similarity decreases as the difference between the features of the ROI and the features of the predetermined request increases.

[0035] In this document, "zero-learning-based information processing apparatus" refers to an apparatus that uses an activation heatmap obtained by the information processing apparatus 100 to identify unlearned video frames to determine the spatial location where a predetermined request occurs, and uses a temporal similarity sequence to identify unlearned frames to determine whether they correspond to a predetermined request.

[0036] For example, a user may typically want to watch a video segment that they specifically desire. In this case, it is necessary to identify the desired frame from the video. As described above, the information processing apparatus 100 according to embodiments of this disclosure can obtain a similarity time series that can be used to identify target frames in a video corresponding to a predetermined request, thereby enabling, for example, a user to use the similarity time series to identify and subsequently watch the desired video segment.

[0037] Since frames typically involve multiple objects, states, and / or events, objects, states, and / or events other than those corresponding to the predetermined request may interfere with the identification of the target frame. As described above, the information processing apparatus 100 according to embodiments of the present disclosure can determine the region of interest of a frame based on the predetermined request and calculate the similarity between the region of interest of the frame and the predetermined request. Compared with a first method that directly calculates the similarity between the frame and the predetermined request, the accuracy of similarity calculation can be improved. Therefore, the similarity time series obtained by the information processing apparatus 100 according to embodiments of the present disclosure can more accurately identify the target frame corresponding to the predetermined request compared with the similarity time series obtained using the first method.

[0038] Figure 4A and Figure 4B Examples of similarity time series obtained using the first method and the information processing apparatus 100 according to embodiments of the present disclosure are shown respectively. Figure 4A and Figure 4B In the diagram, the horizontal axis represents the frame number in the example video, the vertical axis represents the similarity between that frame and the pre-defined request, and the double dashes represent the truth value (i.e., the actual frame number corresponding to the pre-defined request). Figure 4A As shown, the peak value of the similarity time series obtained using the first method is not within the true range, which may lead to the target frame identified based on this similarity time series not being the actual frame corresponding to the predetermined request. On the other hand, as... Figure 4B As shown, the peak value of the similarity time series obtained by the information processing apparatus 100 according to an embodiment of the present disclosure is within the true value range, which makes it possible to accurately identify the actual frame corresponding to the predetermined request based on the similarity time series.

[0039] Figure 5 Examples are shown of the results of target frame identification using similarity time series obtained by the first method and the information processing apparatus 100 according to embodiments of the present disclosure, respectively. Figure 5In this context, R@1IoU = 0.5 represents the recall rate if the IoU (Intersection over Union) between the optimal recognition result and the ground truth is greater than 0.5, thus determining the recognition result as correct. R@1IoU = 0.7 represents the recall rate if the IoU between the optimal recognition result and the ground truth is greater than 0.7, thus determining the recognition result as correct. A Duration of 0.4 indicates that in the recognition process, frames centered at the peak of the similarity time series with an interval of 0.4*V are identified as target frames, where V is a positive natural number representing the total number of frames in the corresponding video. For example... Figure 5 As shown, for R@1IoU=0.7, the information processing device 100 improves upon the first method by approximately 1.72%.

[0040] Furthermore, the information processing apparatus 100 according to embodiments of this disclosure can determine the region of interest of each frame and obtain a similarity time series based on a predetermined request, so that the number of the target frame corresponding to the predetermined request and the image region (i.e., region of interest) in the target frame corresponding to the predetermined request can be identified. For example, this makes it possible to detect the time when the object, state and / or event involved in the predetermined request occurs (i.e., the time corresponding to the target frame) and the position of the object, state and / or event in the image space (i.e., the position of the region of interest in the target frame).

[0041] Furthermore, the information processing apparatus 100 according to embodiments of the present disclosure can utilize existing pre-trained multimodal models or a pre-trained multimodal model and a pre-trained attention network to obtain similarity time series, thereby omitting the training process.

[0042] Furthermore, as described above, event detection can be performed using the similarity time series acquired by the information processing apparatus 100 according to embodiments of the present disclosure. For example, this top-down detection mechanism enables the identification of relationships between objects.

[0043] For example, according to embodiments of this disclosure, determining the region of interest (ROI) of a frame may include using an iterative search method or a grid search method to determine the ROI of the frame based on the frame's activation heatmap.

[0044] For example, according to embodiments of this disclosure, determining the region of interest (ROI) of a frame using a grid search method may include: setting multiple possible values ​​for each parameter in a parameter group used to define the ROI box; obtaining multiple combinations of possible values ​​for the parameter group based on the possible values ​​of each parameter; generating multiple candidate ROI boxes based on the multiple combinations of possible values; selecting the optimal candidate ROI box from the multiple candidate ROI boxes as the ROI box; and determining the ROI based on the ROI box. For example, the region in the frame corresponding to the ROI box can be determined as the ROI.

[0045] By using a grid search method to determine the region of interest as described above, multiple candidate region of interest boxes can be generated in parallel, thereby improving processing speed, for example.

[0046] As an example, selecting the optimal candidate region of interest (ROI) box includes one of the following: selecting the candidate ROI box with the highest similarity between its determined ROI and the frame from among multiple candidate ROI boxes; selecting the candidate ROI box with the highest similarity between the features of its determined ROI and the features of a predetermined request from among multiple candidate ROI boxes; and selecting the candidate ROI box with the highest similarity between the activation heatmap of its determined ROI and the activation heatmap of the frame from among multiple candidate ROI boxes.

[0047] For example, the region of interest (ROI) box can be circular, elliptical, or a polygon such as a rectangle, and those skilled in the art can select an appropriate shape for the ROI box according to actual needs. For instance, the shape of the ROI box can be set based on the activation heatmap of the frame.

[0048] Accordingly, the parameters included in the parameter set can differ depending on the shape of the region of interest (ROI). For example, if the ROI is square, the parameter set may include the center coordinates and side length of the ROI. Alternatively, if the ROI is rectangular, the parameter set may include the center coordinates, aspect ratio, and length or width. Furthermore, if the ROI is circular, the parameter set may include the center coordinates and radius of the ROI.

[0049] As an example, the possible values ​​for each parameter can be set based on a predetermined range and in predetermined steps.

[0050] As another example, the possible values ​​of at least one parameter can be set based on the frame's activation heatmap. For instance, the frame's activation heatmap can be discretized (e.g., max pooling) or segmented, and the coordinates of one or more local extrema (also referred to as "discrete centers") in the processed frame's activation heatmap can be used as the possible values ​​for the center coordinates of the frame's region of interest bounding box. Furthermore, for example, the possible values ​​of parameters other than the center coordinates can be determined based on the distribution of discretized points around the local extrema in the processed frame's activation heatmap. By setting the possible values ​​of at least one parameter based on the frame's activation heatmap, the number of possible values ​​for the aforementioned at least one parameter can be reduced, thereby improving processing speed.

[0051] For example, when using a square region of interest bounding box, an aspect ratio parameter can be introduced for image regions that are far from the center of the frame. For instance, for a combination of possible values ​​in the parameter set where the difference between the center coordinates and the center coordinates of the frame is greater than or equal to a predetermined value, an aspect ratio parameter can be introduced. This allows for a more accurate determination of the region of interest, thereby enabling a more accurate calculation of the similarity between the frame and the predetermined request.

[0052] Figure 6 This illustrates the similarity between a frame containing an event corresponding to a predetermined request and the predetermined request, calculated using the first method and the information processing apparatus 100. For example... Figure 6 As shown, when only a square region of interest box is used, the similarity calculated by the information processing device 100 is about 12% higher than the similarity calculated by the first method; when a square region of interest box and a rectangular region of interest box are used in combination (i.e., the aspect ratio parameter is introduced as described above), the similarity calculated by the information processing device 100 can be further improved.

[0053] Furthermore, since the square region of interest has fewer parameters than the rectangular region of interest, combining square and rectangular regions of interest can further reduce the processing load compared to using only rectangular regions of interest.

[0054] For example, according to embodiments of this disclosure, determining the region of interest (ROI) of a frame using an iterative search method may include: setting an initial set of parameter groups for defining the ROI box; performing an iterative search based on the initial set of parameter groups to achieve a second predetermined condition, thereby obtaining a final set of parameter groups; and generating a ROI box based on the final set of parameter groups, and determining the ROI based on the ROI box.

[0055] For example, the initial value of at least one parameter in the initial value group can be set based on the frame's activation heatmap. For instance, if the parameter group includes center coordinates, the frame's activation heatmap can be discretized (e.g., maxpooling) or segmented, and the coordinates of the local extremum point with the largest value among one or more local extrema in the processed frame's activation heatmap can be used as the initial value for the center coordinates. Furthermore, for example, the initial values ​​of parameters other than the center coordinates can be determined based on the distribution of discretized points around the local extrema in the processed frame's activation heatmap. Setting the initial value of at least one parameter based on the frame's activation heatmap can, for example, improve processing speed.

[0056] As an example, the second predetermined condition can be one of the following conditions (1) to (5). Condition (1): The iteration reaches a predetermined number of times N1; Condition (2): The similarity between the region of interest (ROI) of the frame determined based on the ROI box and the frame is greater than or equal to a fourth predetermined threshold; Condition (3): The similarity between the features of the ROI extracted by the pre-trained multimodal model and the features of the predetermined request is greater than or equal to a fifth predetermined threshold; Condition (4): The similarity between the activation heatmap of the aforementioned ROI and the activation heatmap of the frame is greater than or equal to a sixth predetermined threshold; and Condition (5): The difference between the ROI box and the ROI box of the previous iteration is less than or equal to a seventh predetermined threshold. Those skilled in the art can set appropriate predetermined number of times N1, fourth predetermined threshold, fifth predetermined threshold, sixth predetermined threshold and / or seventh predetermined threshold according to actual needs.

[0057] For example, as those skilled in the art will understand, if the second predetermined condition is one of the conditions (2) to (5) above, and the second predetermined condition is still not met after the predetermined number of iterations N2, the initial group can be reset and the iterative search can be performed again. Those skilled in the art can set an appropriate predetermined number of iterations N2 according to actual needs.

[0058] For example, an iterative search can be performed using the Expectation-Maximum (EM) algorithm or a similar algorithm.

[0059] The above describes an example of determining the region of interest (ROI) of each frame using an iterative search method or a grid search method. However, in some embodiments of this disclosure, if the current frame is not the first frame in the video and a first predetermined condition is met, the ROI of the current frame can be determined based on the ROI of the previous frame. The first predetermined condition can be one of the following: the distance between at least one of the estimated centers of the activation heatmap of the current frame and the center of the ROI of the previous frame is less than or equal to a first predetermined threshold; the difference between the features of the current frame extracted via a multimodal model and the features of the previous frame is less than or equal to a second predetermined threshold; and the difference between the features of the current frame extracted via a multimodal model and the features of the ROI of the previous frame is less than or equal to a third predetermined threshold. Those skilled in the art can set appropriate first, second, and / or third predetermined thresholds according to actual needs.

[0060] By determining the region of interest (ROI) of the current frame based on the ROI of the previous frame, as described above, when the current frame is not the first frame in the video and the first predetermined condition is met, the processing speed can be improved.

[0061] For example, the activation heatmap of the current frame can be discretized or segmented, and one or more local extrema of the processed activation heatmap can be used as the estimation center of the activation heatmap of the current frame.

[0062] As an example, determining the region of interest (ROI) of the current frame based on the ROI of the previous frame may include: selecting an image frame corresponding to the ROI of the previous frame as a first ROI frame, and determining the ROI of the current frame based on the first ROI frame.

[0063] As another example, determining the region of interest (ROI) of the current frame based on the ROI of the previous frame may include: selecting an image frame corresponding to the ROI of the previous frame as a first ROI frame; obtaining a parameter set for a second ROI frame based on the parameter set of the first ROI frame; determining the ROI of the frame as the first ROI and the second ROI frame respectively based on the first and second ROI frames; and determining the ROI of the first and second ROI frames that has a greater similarity to a predetermined request as the ROI of the current frame. By introducing a second ROI frame as described above, and determining the ROI of the current frame that has a greater similarity to a predetermined request between the first ROI frame corresponding to the first ROI frame and the second ROI frame corresponding to the second ROI frame as the ROI of the current frame, the ROI of the current frame can be determined more accurately.

[0064] For example, if the current frame is the second frame in a video, obtaining the parameter set of the second region of interest (ROI) based on the parameter set of the first ROI may include c) or d): c) processing the parameter set of the first ROI using a state estimation algorithm (such as a Kalman filter) to obtain the parameter set of the second ROI; d) performing a predetermined translation operation and / or a predetermined scaling operation on the first ROI to obtain the parameter set of the second ROI. Those skilled in the art can set the predetermined translation operation and / or the predetermined scaling operation according to actual needs.

[0065] For example, when the current frame is the nth or greater than the 3rd frame in the video, the parameter set for obtaining the second region of interest (ROI) based on the parameter set of the first ROI includes one of c), d), and e) above: e) Based on the positional and dimensional relationships between the ROIs of the previous m frames, the first ROI is translated and / or scaled to obtain the parameter set of the second ROI, where m is a natural number greater than 1 and m < n. For example, if the size of the ROIs of the previous m frames increases with the frame number, the size of the first ROI can be increased by a proportion determined based on the increasing trend of the ROIs of the previous m frames, thereby obtaining the parameter set of the second ROI. Furthermore, for example, if the center coordinates of the ROIs of the previous m frames are translated with the increasing frame number, the first ROI can be translated by a degree of translation determined based on the translation trend of the ROIs of the previous m frames, thereby obtaining the parameter set of the second ROI.

[0066] Figure 7 This is a block diagram illustrating an example functional configuration of a zero-learning-based information processing apparatus 700 according to another embodiment of the present disclosure. Figure 7 As shown, the zero-learning-based information processing apparatus 700 according to another embodiment of the present disclosure may include an activation heatmap generation unit 702, a spatial region of interest determination unit 704, a similarity time series acquisition unit 706, a target frame localization unit 708, and a video generation unit 710. The activation heatmap generation unit 702, the spatial region of interest determination unit 704, and the similarity time series acquisition unit 706 are similar to the activation heatmap generation unit 102, the spatial region of interest determination unit 104, and the similarity time series acquisition unit 106 included in the information processing apparatus 100 according to the embodiments of the present disclosure described above; therefore, specific details will not be repeated.

[0067] The target frame localization unit 708 can be configured to identify target frames in a video based on a similarity time series. For example, the target frame localization unit 708 can identify the frame corresponding to the peak of the similarity time series and frames near the peak as target frames. For instance, the target frame localization unit 708 can identify frames centered on the frame corresponding to the peak of the similarity time series with an interval of Duration*V as target frames, where V is a natural number greater than 0, representing the total number of frames contained in the corresponding video. Those skilled in the art can select the method for setting Duration according to actual needs, or they can directly specify a fixed Duration.

[0068] The video generation unit 710 can be configured to generate a new video based on the region of interest determined by the spatial region of interest determination unit 704 of the target frame, so that, for example, a user can directly watch the part of the video that corresponds to the user's predetermined request.

[0069] Furthermore, the new video generated by the video generation unit 710 may include only the region of interest of the target frame, which may reduce the size of the new video, for example.

[0070] Note that in Figure 7 The video generation unit 710 is shown in a dashed box to indicate that in some embodiments, the information processing apparatus 700 may not include the video generation unit 710.

[0071] The information processing apparatus according to embodiments of the present disclosure has been described above. Corresponding to the embodiments of the information processing apparatus described above, the present disclosure also provides embodiments of the following information processing methods.

[0072] Figure 8 This is a flowchart illustrating an example of a zero-learning-based information processing method 800 according to an embodiment of the present disclosure. Figure 8 As shown, the information processing method according to the embodiments of this disclosure may begin at a start step S802 and end at a finish step S816, and may include an activation heatmap generation step S804, a spatial region of interest determination step S806, and a similarity time series acquisition step S808.

[0073] In the activation heatmap generation step S804, an activation heatmap can be generated for each frame in the video based on a predetermined request using a pre-trained multimodal model or a pre-trained attention network. For example, the activation heatmap generation step S804 can be implemented by the activation heatmap generation unit 102 in the information processing apparatus 100 according to the embodiments of this disclosure. Therefore, for specific details, please refer to the description of the activation heatmap generation unit 102 above, which will not be repeated here.

[0074] As an example, a reservation request may include at least one of text, images, and voice, but is not limited to these.

[0075] In the spatial region of interest determination step S806, the region of interest for each frame can be determined based on the activation heatmap of that frame. For example, the spatial region of interest determination step S806 can be implemented by the spatial region of interest determination unit 104 in the information processing apparatus 100 according to the embodiments of this disclosure. Therefore, for specific details, please refer to the description of the spatial region of interest determination unit 104 above, and it will not be repeated here.

[0076] In the similarity time series acquisition step S808, a pre-trained multimodal model can be used to calculate the similarity between the region of interest of each frame and the predetermined request to obtain the similarity time series of the video. This similarity time series can be used to identify the target frame corresponding to the predetermined request in the video. For example, the similarity time series acquisition step S808 can be implemented by the similarity time series acquisition unit 106 in the information processing apparatus 100 according to the embodiments of this disclosure. Therefore, for specific details, please refer to the description of the similarity time series acquisition unit 106 above, which will not be repeated here.

[0077] Similar to the information processing apparatus 100 according to an embodiment of the present disclosure, the information processing method 800 according to an embodiment of the present disclosure can obtain a similarity time series that can be used to identify target frames corresponding to a predetermined request in a video, thereby enabling, for example, a user to use the similarity time series to identify and then watch the video segment they desire.

[0078] Furthermore, the information processing method 800 according to embodiments of this disclosure can determine the region of interest (ROI) of a frame based on a predetermined request and calculate the similarity between the ROI of the frame and the predetermined request. Compared with the first method described above, the accuracy of the similarity calculation can be improved. Therefore, the similarity time series obtained by the information processing method 800 according to embodiments of this disclosure can more accurately identify the target frame corresponding to the predetermined request compared with the similarity time series obtained using the first method.

[0079] In addition, the information processing method 800 according to the embodiments of this disclosure can determine the region of interest of each frame and obtain a similarity time series based on a predetermined request, so that the number of the target frame corresponding to the predetermined request and the image region in the target frame corresponding to the predetermined request can be identified. For example, this makes it possible to detect the time when the object, state and / or event involved in the predetermined request occurs and the position of the object, state and / or event in the image space.

[0080] Furthermore, the information processing method 800 according to the embodiments of this disclosure can utilize existing pre-trained multimodal models or a pre-trained multimodal model and a pre-trained attention network to obtain similarity time series, thereby omitting the training process.

[0081] Furthermore, similar to the information processing apparatus 100 according to embodiments of the present disclosure, event detection can be performed using the similarity time series obtained by the information processing method 800 according to embodiments of the present disclosure. For example, this top-down detection mechanism enables the identification of relationships between objects.

[0082] For example, according to an embodiment of this disclosure, in the spatial region of interest determination step S806, an iterative search method or a grid search method can be used to determine the region of interest of a frame based on the activation heatmap of the frame.

[0083] For example, according to embodiments of this disclosure, determining the region of interest (ROI) of a frame using a grid search method may include: setting multiple possible values ​​for each parameter in a parameter group used to define the ROI box; obtaining multiple combinations of possible values ​​for the parameter group based on the possible values ​​of each parameter; generating multiple candidate ROI boxes based on the multiple combinations of possible values; selecting the optimal candidate ROI box from the multiple candidate ROI boxes as the ROI box; and determining the ROI based on the ROI box. For example, the region in the frame corresponding to the ROI box can be determined as the ROI.

[0084] By using a grid search method to determine the region of interest as described above, multiple candidate region of interest boxes can be generated in parallel, thereby improving processing speed, for example.

[0085] As an example, selecting the optimal candidate region of interest (ROI) box includes one of the following: selecting the candidate ROI box with the highest similarity between its determined ROI and the frame from among multiple candidate ROI boxes; selecting the candidate ROI box with the highest similarity between the features of its determined ROI and the features of a predetermined request from among multiple candidate ROI boxes; and selecting the candidate ROI box with the highest similarity between the activation heatmap of its determined ROI and the activation heatmap of the frame from among multiple candidate ROI boxes.

[0086] As an example, the possible values ​​for each parameter can be set based on a predetermined range and in predetermined steps.

[0087] As another example, the possible values ​​of at least one parameter can be set based on the frame's activation heatmap. For example, the frame's activation heatmap can be discretized (e.g., max pooling) or segmented, and the coordinates of one or more local extrema of the processed frame's activation heatmap can be used as the possible values ​​for the center coordinates of the frame's region of interest bounding box. Furthermore, for example, the possible values ​​of parameters other than the center coordinates can be determined based on the distribution of discretized points around the local extrema of the processed frame's activation heatmap. By setting the possible values ​​of at least one parameter based on the frame's activation heatmap, the number of possible values ​​for that at least one parameter can be reduced, thereby improving processing speed.

[0088] For example, according to embodiments of this disclosure, determining the region of interest (ROI) of a frame using an iterative search method may include: setting an initial set of parameter groups for defining the ROI box; performing an iterative search based on the initial set of parameter groups to achieve a second predetermined condition, thereby obtaining a final set of parameter groups; and generating a ROI box based on the final set of parameter groups, and determining the ROI based on the ROI box.

[0089] For example, the initial value of at least one parameter in the initial value group can be set based on the frame's activation heatmap. For instance, if the parameter group includes center coordinates, the frame's activation heatmap can be discretized (e.g., maxpooling) or segmented, and the coordinates of the local extremum point with the largest value among one or more local extrema in the processed frame's activation heatmap can be used as the initial value for the center coordinates. Furthermore, for example, the initial values ​​of parameters other than the center coordinates can be determined based on the distribution of discretized points around the local extrema in the processed frame's activation heatmap. Setting the initial value of at least one parameter based on the frame's activation heatmap can, for example, improve processing speed.

[0090] As an example, the second predetermined condition can be one of the following conditions (1) to (5). Condition (1): The iteration reaches a predetermined number of times N1; Condition (2): The similarity between the region of interest (ROI) of the frame determined based on the ROI box and the frame is greater than or equal to a fourth predetermined threshold; Condition (3): The similarity between the features of the ROI extracted by the pre-trained multimodal model and the features of the predetermined request is greater than or equal to a fifth predetermined threshold; Condition (4): The similarity between the activation heatmap of the aforementioned ROI and the activation heatmap of the frame is greater than or equal to a sixth predetermined threshold; and Condition (5): The difference between the ROI box and the ROI box of the previous iteration is less than or equal to a seventh predetermined threshold. Those skilled in the art can set appropriate predetermined number of times N1, fourth predetermined threshold, fifth predetermined threshold, sixth predetermined threshold and / or seventh predetermined threshold according to actual needs.

[0091] For example, as those skilled in the art will understand, if the second predetermined condition is one of the conditions (2) to (5) above, and the second predetermined condition is still not met after the predetermined number of iterations N2, the initial group can be reset and the iterative search can be performed again. Those skilled in the art can set an appropriate predetermined number of iterations N2 according to actual needs.

[0092] For example, an iterative search can be performed using the expectation-maximization algorithm or similar algorithms.

[0093] The above describes an example of determining the region of interest (ROI) of each frame using an iterative search method or a grid search method. However, in some embodiments of this disclosure, if the current frame is not the first frame in the video and a first predetermined condition is met, the ROI of the current frame can be determined based on the ROI of the previous frame. The first predetermined condition can be one of the following: the distance between at least one of the estimated centers of the activation heatmap of the current frame and the center of the ROI of the previous frame is less than or equal to a first predetermined threshold; the difference between the features of the current frame extracted via a multimodal model and the features of the previous frame is less than or equal to a second predetermined threshold; and the difference between the features of the current frame extracted via a multimodal model and the features of the ROI of the previous frame is less than or equal to a third predetermined threshold. Those skilled in the art can set appropriate first, second, and / or third predetermined thresholds according to actual needs.

[0094] By determining the region of interest (ROI) of the current frame based on the ROI of the previous frame, as described above, when the current frame is not the first frame in the video and the first predetermined condition is met, the processing speed can be improved.

[0095] For example, the activation heatmap of the current frame can be discretized or segmented, and one or more local extrema of the processed activation heatmap can be used as the estimation center of the activation heatmap of the current frame.

[0096] For example, according to embodiments of this disclosure, determining the region of interest (ROI) of the current frame based on the ROI of the previous frame may include: selecting an image frame corresponding to the ROI of the previous frame as a first ROI frame; obtaining a parameter set of a second ROI frame based on the parameter set of the first ROI frame; extracting the ROI of the frame based on the first ROI frame and the second ROI frame as the first ROI and the second ROI, respectively; and determining the ROI of the first ROI and the second ROI that has a greater similarity to a predetermined request as the ROI of the current frame.

[0097] For example, if the current frame is the second frame in a video, obtaining the parameter set of the second region of interest (ROI) based on the parameter set of the first ROI may include c) or d): c) processing the parameter set of the first ROI using a state estimation algorithm (such as a Kalman filter) to obtain the parameter set of the second ROI; d) performing a predetermined translation operation and / or a predetermined scaling operation on the first ROI to obtain the parameter set of the second ROI. Those skilled in the art can set the predetermined translation operation and / or the predetermined scaling operation according to actual needs.

[0098] For example, when the current frame is the nth or greater than the 3rd frame in the video, the parameter set for obtaining the second region of interest (ROI) based on the parameter set of the first ROI includes one of c), d), and e) above: e) Based on the positional and dimensional relationships between the ROIs of the previous m frames, the first ROI is translated and / or scaled to obtain the parameter set of the second ROI, where m is a natural number greater than 1. For example, if the size of the ROIs of the previous m frames increases with the frame number, the size of the first ROI can be increased by a proportion determined based on the increasing trend of the ROIs of the previous m frames, thereby obtaining the parameter set of the second ROI. Furthermore, for example, if the center coordinates of the ROIs of the previous m frames are translated with the increasing frame number, the first ROI can be translated by a degree determined based on the translation trend of the ROIs of the previous m frames, thereby obtaining the parameter set of the second ROI.

[0099] Figure 9 This is a flowchart illustrating an example of a zero-learning-based information processing method 900 according to another embodiment of the present disclosure. Figure 9 As shown, the information processing method 900 according to another embodiment of this disclosure may begin at a start step S902 and end at an end step S916, and may include an activation heatmap generation step S904, a spatial region of interest determination step S906, a similarity time series acquisition step S908, a target frame localization step S910, and a video generation step S912. The activation heatmap generation step S904, the spatial region of interest determination step S906, and the similarity time series acquisition step S908 are similar to the activation heatmap generation step S804, the spatial region of interest determination step S806, and the similarity time series acquisition step S808 included in the information processing method 800 according to the embodiment of this disclosure described above; therefore, specific details will not be repeated.

[0100] In the target frame localization step S910, the target frame can be identified in the video based on the similarity time series of the video. For example, the target frame localization step S910 can be implemented by the target frame localization unit 708 in the information processing apparatus 700 according to the embodiments of the present disclosure. Therefore, for specific details, please refer to the description of the target frame localization unit 708 above, which will not be repeated here.

[0101] In video generation step S912, a new video can be generated based on the region of interest determined in step S906 via spatial region of interest determination of the target frame, so that, for example, a user can directly watch the part of the video corresponding to the user's predetermined request. For example, video generation step S912 can be implemented by video generation unit 710 in the information processing apparatus 700 according to the embodiments of this disclosure described above. Therefore, for specific details, please refer to the description of video generation unit 710 above, which will not be repeated here.

[0102] Note that in Figure 9 The video generation step S912 is shown in a dashed box to indicate that in some embodiments, the information processing method 900 may not include the video generation step S912.

[0103] It should be noted that although the functional configuration and operation of the information processing apparatus and information processing method according to the embodiments of the present disclosure have been described above, these are merely examples and not limitations. Those skilled in the art can modify the above embodiments based on the principles of the present disclosure, such as adding, deleting or combining functional modules and operations in various embodiments, and all such modifications fall within the scope of the present disclosure.

[0104] Furthermore, it should be noted that the method embodiments described here correspond to the apparatus embodiments described above. Therefore, any content not described in detail in the method embodiments can be found in the description of the corresponding parts in the apparatus embodiments, and will not be repeated here.

[0105] Furthermore, this disclosure also provides a storage medium and a program product. It should be understood that the machine-executable instructions in the storage medium and program product according to embodiments of this disclosure can also be configured to perform the aforementioned information processing methods; therefore, details not described in detail here can be referred to the descriptions in the preceding corresponding sections and will not be repeated here.

[0106] Accordingly, the storage medium used to carry the aforementioned program product including machine-executable instructions is also included in the disclosure of this invention. This storage medium includes, but is not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0107] Furthermore, it should be noted that the aforementioned series of processes and devices can also be implemented via software and / or firmware. In the case of software and / or firmware implementation, data can be transferred from storage media or networks to computers with dedicated hardware architectures, such as… Figure 10 The general-purpose personal computer 1000 shown is equipped with the programs that constitute the software, and when various programs are installed, the computer is able to perform various functions, etc.

[0108] exist Figure 10 In this system, the central processing unit (CPU) 1001 performs various processes based on the program stored in the read-only memory (ROM) 1002 or the program loaded into the random access memory (RAM) 1003 from the storage section 1008. The RAM 1003 also stores, as needed, the data required when the CPU 1001 performs various processes.

[0109] CPU 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output interface 1005 is also connected to bus 1004.

[0110] The following components are connected to the input / output interface 1005: input section 1006, including a keyboard, mouse, etc.; output section 1007, including a display, such as a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 1008, including a hard disk, etc.; and communication section 1009, including a network interface card, such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network, such as the Internet.

[0111] As needed, drive 1010 is also connected to input / output interface 1005. Removable media 1011, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 1010 as needed, so that computer programs read from them can be installed into storage section 1008 as needed.

[0112] When the above series of processes are implemented through software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable media 1011.

[0113] Those skilled in the art will understand that such storage media are not limited to Figure 10The illustration shows a removable medium 1011 containing a program, distributed separately from the device to provide the program to the user. Examples of removable media 1011 include magnetic disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-disk (MD) (registered trademark)), and semiconductor memory. Alternatively, the storage medium may be ROM 1002, a hard disk included in storage section 1008, etc., containing programs and distributed to the user along with the device containing them.

[0114] Preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but the present disclosure is by no means limited to the examples described above. Various changes and modifications can be made by those skilled in the art within the scope of the appended claims, and it should be understood that such changes and modifications naturally fall within the technical scope of the present disclosure.

[0115] For example, the multiple functions included in one unit in the above embodiments can be implemented by separate devices. Alternatively, the multiple functions implemented by multiple units in the above embodiments can be implemented by separate devices respectively. In addition, one of the above functions can be implemented by multiple units. Needless to say, such a configuration is included within the scope of the present disclosure.

[0116] In this specification, the steps described in the flowchart include not only processes executed sequentially in the stated order, but also processes executed in parallel or individually, rather than necessarily sequentially. Furthermore, even within the steps of sequential processing, needless to say, the order can be appropriately altered.

[0117] In addition, the technology disclosed herein can also be configured as follows.

[0118] Appendix 1. An information processing device, comprising:

[0119] An activation heatmap generation unit is configured to generate an activation heatmap for each frame in the video, using a pre-trained multimodal model or a pre-trained attention network, based on a predetermined request.

[0120] A spatial region of interest (ROI) determination unit is configured to determine the ROI of a frame for each frame based on the frame's activation heatmap; and

[0121] The similarity time series acquisition unit is configured to use a pre-trained multimodal model to calculate the similarity between the region of interest of each frame and the predetermined request, so as to obtain the similarity time series of the video, which can be used to identify the target frame corresponding to the predetermined request in the video.

[0122] Appendix 2. The information processing apparatus according to Appendix 1 further includes:

[0123] A target frame localization unit is configured to identify the target frame in the video based on a similarity time series of the video; and

[0124] The video generation unit is configured to generate a new video based on the region of interest of the target frame determined by the spatial region of interest determination unit.

[0125] Appendix 3. According to the information processing apparatus described in Appendix 1, determining the region of interest of the frame includes a) or (b):

[0126] a) Using an iterative search method or a grid search method, determine the region of interest (ROI) of the frame based on the activation heatmap of the frame.

[0127] (b) If the frame is the first frame in the video or does not meet the first predetermined condition, determine the region of interest (ROI) of the frame based on the activation heatmap of the frame using the iterative search method or the grid search method; and if the frame is not the first frame in the video but meets the first predetermined condition, determine the ROI of the frame based on the ROI of the previous frame.

[0128] Wherein, the first predetermined condition is one of the following: the distance between at least one of the estimated centers of the activation heatmap of the frame and the center of the region of interest of the previous frame is less than or equal to a first predetermined threshold; the difference between the features of the frame extracted by the pre-trained multimodal model and the features of the previous frame is less than or equal to a second predetermined threshold; and the difference between the features of the frame extracted by the pre-trained multimodal model and the features of the region of interest of the previous frame is less than or equal to a third predetermined threshold.

[0129] The estimation center is one or more local extrema points of the activation heatmap of the frame after discretization or segmentation.

[0130] Appendix 4. The information processing apparatus according to Appendix 3, wherein determining the region of interest of the frame based on the region of interest of the previous frame includes:

[0131] Select the image frame corresponding to the region of interest in the previous frame as the first region of interest frame;

[0132] The parameter set of the second region of interest is obtained based on the parameter set of the first region of interest box;

[0133] The regions of interest (ROIs) of the frame are determined as the first ROI and the second ROI based on the first ROI bounding box and the second ROI bounding box, respectively; and

[0134] The region of interest with greater similarity to the predetermined request between the first region of interest and the second region of interest is determined as the region of interest of the frame.

[0135] Appendix 5. According to the information processing apparatus described in Appendix 4, when the frame is the second frame in the video, obtaining the parameter set of the second region of interest box based on the parameter set of the first region of interest box includes c) or d):

[0136] c) The parameter set of the first region of interest box is processed using a state estimation algorithm to obtain the parameter set of the second region of interest box.

[0137] d) Perform a predetermined translation operation and / or a predetermined scaling operation on the first region of interest box to obtain the parameter set of the second region of interest box, and

[0138] When the frame is the nth or greater than the 3rd frame in the video, the parameter set for obtaining the second region of interest (ROI) based on the parameter set of the first ROI includes one of c), d), and e):

[0139] e) Based on the positional and dimensional relationships between the regions of interest in the first m frames of the frame, translate and / or scale the first region of interest box to obtain the parameter set of the second region of interest box, where m is a natural number greater than 1 and m < n.

[0140] Appendix 6. The information processing apparatus according to any one of Appendices 3 to 5, wherein determining the region of interest of the frame using the grid search method comprises:

[0141] For each parameter in the parameter group used to define the region of interest bounding box, set multiple possible values ​​for that parameter;

[0142] Based on the possible values ​​of each parameter in the parameter group, obtain multiple combinations of possible values ​​for the parameter group; and

[0143] Multiple candidate region of interest (ROI) boxes are generated based on the multiple possible value combinations. The optimal ROI box is selected from the multiple ROI boxes as the ROI box, and the ROI is determined based on the ROI box.

[0144] Note 7. In the information processing apparatus according to Note 6, selecting the optimal candidate region of interest box includes one of the following:

[0145] The candidate region of interest (ROI) box with the highest similarity between its determined ROI and the frame is selected from the plurality of candidate ROI boxes as the optimal candidate ROI box.

[0146] The candidate region of interest (ROI) box with the highest similarity between the features of its determined ROI and the features of the predetermined request is selected from the plurality of candidate ROI boxes as the optimal candidate ROI box; and

[0147] The candidate region of interest (ROI) box with the highest similarity between the activation heatmap of its determined ROI and the activation heatmap of the frame is selected as the optimal candidate ROI box.

[0148] Appendix 8. In the information processing apparatus according to any one of Appendices 3 to 5, determining the region of interest of the frame using the iterative search method includes:

[0149] An initial set of parameter values ​​is set to define the bounding box of interest; an iterative search is performed based on the initial set of parameter values ​​to achieve a second predetermined condition, thereby obtaining the final set of parameter values; and

[0150] The region of interest (ROI) bounding box is generated based on the final value set, and the ROI is determined based on the ROI bounding box.

[0151] The initial value group is set based on the activation heatmap of the frame.

[0152] Note 9. The information processing apparatus according to any one of Notes 1 to 5, wherein the predetermined request includes at least one of text, voice, and image.

[0153] Appendix 10. An information processing method, comprising:

[0154] The activation heatmap generation step is used to generate an activation heatmap for each frame in the video based on a predetermined request, using a pre-trained multimodal model or a pre-trained attention network.

[0155] The spatial region of interest determination step is used to determine the region of interest for each frame based on the frame's activation heatmap; and

[0156] The similarity time series acquisition step is used to calculate the similarity between the region of interest of each frame and the predetermined request using a pre-trained multimodal model, so as to obtain the similarity time series of the video, which can be used to identify the target frame corresponding to the predetermined request in the video.

[0157] Note 11. The information processing method according to Note 10 further includes:

[0158] The target frame localization step is used to identify the target frame in the video based on the similarity time series of the video; and

[0159] A video generation step is used to generate a new video based on the region of interest of the target frame determined via the spatial region of interest determination step.

[0160] Note 12. According to the information processing method described in Note 10, the region of interest of the frame is determined to include a) or (b):

[0161] a) Using an iterative search method or a grid search method, determine the region of interest (ROI) of the frame based on the activation heatmap of the frame.

[0162] (b) If the frame is the first frame in the video or does not meet the first predetermined condition, determine the region of interest (ROI) of the frame based on the activation heatmap of the frame using the iterative search method or the grid search method; and if the frame is not the first frame in the video but meets the first predetermined condition, determine the ROI of the frame based on the ROI of the previous frame.

[0163] Wherein, the first predetermined condition is one of the following: the distance between at least one of the estimated centers of the activation heatmap of the frame and the center of the region of interest of the previous frame is less than or equal to a first predetermined threshold; the difference between the features of the frame extracted by the pre-trained multimodal model and the features of the previous frame is less than or equal to a second predetermined threshold; and the difference between the features of the frame extracted by the pre-trained multimodal model and the features of the region of interest of the previous frame is less than or equal to a third predetermined threshold.

[0164] The estimation center is one or more local extrema points of the activation heatmap of the frame after discretization or segmentation.

[0165] Appendix 13. According to the information processing method described in Appendix 12, determining the region of interest (ROI) of the frame based on the ROI of the previous frame includes:

[0166] Select the image frame corresponding to the region of interest in the previous frame as the first region of interest frame;

[0167] The parameter set of the second region of interest is obtained based on the parameter set of the first region of interest box;

[0168] The regions of interest (ROIs) of the frame are determined as the first ROI and the second ROI based on the first ROI bounding box and the second ROI bounding box, respectively; and

[0169] The region of interest with greater similarity to the predetermined request between the first region of interest and the second region of interest is determined as the region of interest of the frame.

[0170] Appendix 14. According to the information processing method described in Appendix 13, when the frame is the second frame in the video, obtaining the parameter set of the second region of interest (ROI) based on the parameter set of the first ROI includes c) or d):

[0171] c) The parameter set of the first region of interest box is processed using a state estimation algorithm to obtain the parameter set of the second region of interest box.

[0172] d) Perform a predetermined translation operation and / or a predetermined scaling operation on the first region of interest box to obtain the parameter set of the second region of interest box, and

[0173] When the frame is the nth or greater than the 3rd frame in the video, the parameter set for obtaining the second region of interest (ROI) based on the parameter set of the first ROI includes one of c), d), and e):

[0174] e) Based on the positional and dimensional relationships between the regions of interest in the first m frames of the frame, translate and / or scale the first region of interest box to obtain the parameter set of the second region of interest box, where m is a natural number greater than 1 and m < n.

[0175] Note 15. The information processing method according to any one of Notes 12 to 14, wherein, when using the grid search method, determining the region of interest of the frame includes:

[0176] For each parameter in the parameter group used to define the region of interest bounding box, set multiple possible values ​​for that parameter;

[0177] Based on the possible values ​​of each parameter in the parameter group, obtain multiple combinations of possible values ​​for the parameter group; and

[0178] Multiple candidate region of interest (ROI) boxes are generated based on the multiple possible value combinations. The optimal ROI box is selected from the multiple ROI boxes as the ROI box, and the ROI is determined based on the ROI box.

[0179] Note 16. According to the information processing method described in Note 15, selecting the optimal candidate region of interest box includes one of the following:

[0180] The candidate region of interest (ROI) box with the highest similarity between its determined ROI and the frame is selected from the plurality of candidate ROI boxes as the optimal candidate ROI box.

[0181] The candidate region of interest (ROI) box with the highest similarity between the features of its determined ROI and the features of the predetermined request is selected from the plurality of candidate ROI boxes as the optimal candidate ROI box; and

[0182] The candidate region of interest (ROI) box with the highest similarity between the activation heatmap of its determined ROI and the activation heatmap of the frame is selected as the optimal candidate ROI box.

[0183] Note 17. According to any one of Notes 12 to 14, in the case of using the iterative search method, determining the region of interest of the frame includes:

[0184] An initial set of parameter values ​​is set to define the bounding box of interest; an iterative search is performed based on the initial set of parameter values ​​to achieve a second predetermined condition, thereby obtaining the final set of parameter values; and

[0185] The region of interest (ROI) bounding box is generated based on the final value set, and the ROI is determined based on the ROI bounding box.

[0186] The initial value group is set based on the activation heatmap of the frame.

[0187] Note 18. The information processing method according to any one of Notes 10 to 14, wherein the predetermined request includes at least one of text, voice, and image.

[0188] Note 19. A computer-readable storage medium storing a program that, when executed by a computer, causes the computer to perform the method according to any one of Notes 10 to 18.

Claims

1. An information processing device based on zero-learning, comprising: An activation heatmap generation unit is configured to generate an activation heatmap for each frame in the video, using a pre-trained multimodal model or a pre-trained attention network, based on a predetermined request. A spatial region of interest determination unit is configured to determine the region of interest of a frame for each frame based on the frame's activation heatmap; as well as The similarity time series acquisition unit is configured to use a pre-trained multimodal model to calculate the similarity between the region of interest of each frame and the predetermined request, thereby obtaining a similarity time series of the video. This similarity time series can be used to identify the target frame in the video corresponding to the predetermined request. Wherein, determining the region of interest of the frame includes a) or (b): a) Using an iterative search method or a grid search method, determine the region of interest (ROI) of the frame based on the activation heatmap of the frame. (b) If the frame is the first frame in the video or does not meet the first predetermined condition, determine the region of interest (ROI) of the frame based on the activation heatmap of the frame using the iterative search method or the grid search method; and if the frame is not the first frame in the video and meets the first predetermined condition, determine the ROI of the frame based on the ROI of the previous frame. Wherein, the first predetermined condition is one of the following: the distance between at least one of the estimated centers of the activation heatmap of the frame and the center of the region of interest of the previous frame is less than or equal to a first predetermined threshold; the difference between the features of the frame extracted by the pre-trained multimodal model and the features of the previous frame is less than or equal to a second predetermined threshold; and the difference between the features of the frame extracted by the pre-trained multimodal model and the features of the region of interest of the previous frame is less than or equal to a third predetermined threshold. The estimation center is one or more local extrema points of the activation heatmap of the frame after discretization or segmentation.

2. The information processing apparatus according to claim 1, further comprising: A target frame localization unit is configured to identify the target frame in the video based on a similarity time series of the video; as well as The video generation unit is configured to generate a new video based on the region of interest of the target frame determined by the spatial region of interest determination unit.

3. The information processing apparatus according to claim 1, wherein Determining the region of interest (ROI) of a frame based on the ROI of the previous frame includes: Select the image frame corresponding to the region of interest in the previous frame as the first region of interest frame; The parameter set of the second region of interest is obtained based on the parameter set of the first region of interest box; The regions of interest (ROIs) of the frame are determined as the first ROI and the second ROI based on the first ROI bounding box and the second ROI bounding box, respectively; and The region of interest with greater similarity to the predetermined request between the first region of interest and the second region of interest is determined as the region of interest of the frame.

4. The information processing apparatus according to claim 3, wherein when the frame is the second frame in the video, obtaining the parameter set of the second region of interest box based on the parameter set of the first region of interest box includes c) or d): c) The parameter set of the first region of interest box is processed using a state estimation algorithm to obtain the parameter set of the second region of interest box. d) Perform a predetermined translation operation and / or a predetermined scaling operation on the first region of interest box to obtain the parameter set of the second region of interest box, and When the frame is the nth or greater than the 3rd frame in the video, the parameter set for obtaining the second region of interest (ROI) based on the parameter set of the first ROI includes one of c), d), and e): e) translating and / or scaling the first region of interest box based on the positional and dimensional relationships between the regions of interest of the previous m frames of the frame to obtain a parameter set of the second region of interest box, wherein, m is a natural number greater than 1, and m < n.

5. The information processing apparatus according to any one of claims 1 to 4, wherein When using the grid search method, determining the region of interest (ROI) of the frame includes: For each parameter in the parameter group used to define the region of interest bounding box, set multiple possible values ​​for that parameter; Based on the possible values ​​of each parameter in the parameter group, obtain multiple combinations of possible values ​​for the parameter group; and Multiple candidate region of interest (ROI) boxes are generated based on the multiple possible value combinations. The optimal ROI box is selected from the multiple ROI boxes as the ROI box, and the ROI is determined based on the ROI box.

6. The information processing apparatus according to claim 5, wherein The optimal candidate region of interest bounding box includes one of the following: The candidate region of interest (ROI) box with the highest similarity between its determined ROI and the frame is selected from the plurality of candidate ROI boxes as the optimal candidate ROI box. The candidate region of interest (ROI) with the highest similarity between the features of its determined ROI and the features of the predetermined request is selected from among the multiple candidate ROI boxes as the optimal candidate ROI box. as well as The candidate region of interest (ROI) with the highest similarity between its activation heatmap and the activation heatmap of the frame is selected from among the multiple candidate ROI boxes as the optimal candidate ROI box.

7. The information processing apparatus according to any one of claims 1 to 4, wherein determining the region of interest of the frame using the iterative search method comprises: An initial set of parameter values ​​is set to define the region of interest bounding box. An iterative search is performed based on the initial set of parameter values ​​to achieve a second predetermined condition, thereby obtaining the final set of parameter values. as well as The region of interest (ROI) bounding box is generated based on the final value set, and the ROI is determined based on the ROI bounding box. The initial value group is set based on the activation heatmap of the frame.

8. An information processing method based on zero-learning, comprising: The activation heatmap generation step is used to generate an activation heatmap for each frame in the video based on a predetermined request, using a pre-trained multimodal model or a pre-trained attention network. The spatial region of interest determination step is used to determine the region of interest of each frame based on the activation heatmap of the frame. as well as The similarity time series acquisition step is used to calculate the similarity between the region of interest of each frame and the predetermined request using a pre-trained multimodal model, thereby obtaining a similarity time series of the video. This similarity time series can be used to identify the target frame corresponding to the predetermined request in the video. Wherein, determining the region of interest of the frame includes a) or (b): a) Using an iterative search method or a grid search method, determine the region of interest (ROI) of the frame based on the activation heatmap of the frame. (b) If the frame is the first frame in the video or does not meet the first predetermined condition, determine the region of interest (ROI) of the frame based on the activation heatmap of the frame using the iterative search method or the grid search method; and if the frame is not the first frame in the video and meets the first predetermined condition, determine the ROI of the frame based on the ROI of the previous frame. Wherein, the first predetermined condition is one of the following: the distance between at least one of the estimated centers of the activation heatmap of the frame and the center of the region of interest of the previous frame is less than or equal to a first predetermined threshold; the difference between the features of the frame extracted by the pre-trained multimodal model and the features of the previous frame is less than or equal to a second predetermined threshold; and the difference between the features of the frame extracted by the pre-trained multimodal model and the features of the region of interest of the previous frame is less than or equal to a third predetermined threshold. The estimation center is one or more local extrema points of the activation heatmap of the frame after discretization or segmentation.

9. A computer-readable storage medium storing a program that, when executed by a computer, causes the computer to perform the method according to claim 8.

Images