Interactive highlight identification method and device, computer equipment and storage medium

Abstract

The invention relates to an interactive highlight identification method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a panoramic video, wherein the panoramic video is composed of multiple frames of panoramic images containing figures and objects at the same time; aiming at each frame of panoramic image containing the character and the object at the same time, identifying the interaction category of the character and the object; linking the panoramic images with the same interaction category at least based on the interaction category and a preset overlap metric to obtain a linking result; and determining the interactive highlight of the panoramic video based on the external frame in the link result. By adopting the method, the interaction relationship between the person and the object in the panoramic video can be accurately and quickly identified, so that the interaction highlight is determined.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an interactive highlight recognition method, device, computer equipment, and storage medium. Background Art

[0002] With the development of various short video platforms, in order to increase the speed of editing, users usually use various templates adapted from short video templates to edit videos. In panoramic videos, existing panoramic video processing technologies mainly focus on image / video generation, scene understanding, 3D vision, and human-computer interaction.

[0003] However, the technology for automatically capturing highlight moments of the interaction between people and objects in panoramic videos is not yet mature, and it is impossible to accurately and quickly identify the interaction between people and objects in panoramic videos. Summary of the invention

[0004] Based on this, it is necessary to provide an interactive highlight recognition method, device, computer equipment, and storage medium to address the above technical problems.

[0005] In a first aspect, the present disclosure provides an interactive highlight recognition method. The method comprises:

[0006] Acquire a panoramic video, wherein the panoramic video is composed of multiple panoramic images containing both people and objects;

[0007] For each panoramic image frame containing both people and objects, identify the interaction category of the people and objects;

[0008] Based at least on the interaction category and a preset overlap metric, the panoramic images with the same interaction category are linked to obtain a link result;

[0009] Based on the bounding box in the link result, an interactive highlight of the panoramic video is determined.

[0010] In one embodiment, for each panoramic image frame containing both a person and an object, identifying the interaction category of the person and the object includes:

[0011] For each panoramic image frame containing both a person and an object, identifying a person frame of the person and an object frame of the object contained in each panoramic image frame;

[0012] Determining, according to the character frame and the object frame, an interaction point where the character interacts with the object, an interaction category of the interaction point, and an external frame containing the character frame and the object frame;

[0013] The step of linking panoramic images with the same interaction category based at least on the interaction category and a preset overlap metric to obtain a link result includes:

[0014] Based on the interaction points, the interaction categories of the interaction points and a preset overlap metric, the panoramic images indicated by the interaction points with the same interaction category are linked to obtain a link result.

[0015] In one embodiment, determining, based on the character frame and the object frame, the interaction point where the character interacts with the object, the interaction category of the interaction point, and the bounding box containing the character frame and the object frame includes:

[0016] Determine a center point of a person in a person frame and a center point of an object in an object frame;

[0017] Determine an interaction point based on the center point of the character and the center point of the object;

[0018] Based on the character frame and the object frame, generating a bounding box containing the character frame and the object frame;

[0019] Based on the interaction points and the preset interaction categories, generating an interaction heat map including the interaction points and matching the interaction categories;

[0020] The loss is calculated element by element on the interaction heat map to determine the interaction category of the interaction point and the score of the interaction point.

[0021] In one of the embodiments, the circumscribed frame includes the interaction point, and the panoramic images indicated by the interaction points with the same interaction category are linked based on the interaction point, the interaction category of the interaction point and a preset overlap metric to obtain a link result, including:

[0022] According to the scores of the interaction points and the interaction categories of the interaction points, the bounding boxes are sorted and screened in descending order of the scores to obtain a preset number of target bounding boxes;

[0023] Creating a corresponding link for each of the target bounding boxes in the first panoramic image frame;

[0024] Determine each target circumscribed frame that meets a preset connection condition in the panoramic image after the first frame, and sequentially expand the link based on each target circumscribed frame that meets the preset connection condition;

[0025] Among them, each of the target external bounding boxes that meets the preset connection conditions includes: the link is not expanded, the score of the interaction point in the target external bounding box is the highest score, and the average intersection-and-union ratio between the last target external bounding box in the link and the target external bounding box that meets the preset connection conditions is greater than or equal to a preset overlap metric.

[0026] In one embodiment, generating a bounding box containing the character box and the object box based on the character box and the object box includes:

[0027] Based on the character frame and the object frame, generating a bounding rectangle containing the character frame and the object frame, the bounding rectangle being the bounding box;

[0028] Determine a zoom ratio matching each panoramic image frame based on the size of the bounding box in each panoramic image frame;

[0029] The external frame is adjusted according to each panoramic image frame matching the zoom ratio.

[0030] In one of the embodiments, determining the interactive highlight of the panoramic video based on the bounding box in the link result includes:

[0031] Screening is performed based on the interaction categories of the interaction points contained in the bounding box in the link result to determine the bounding box in the link result that meets the preset target interaction category;

[0032] Determining the interactive highlights of the panoramic video according to the bounding box in the link result that meets the target interaction category;

[0033] And / or, based on an average value of the coordinates of the bounding box in the link result, determining the interactive highlights of the panoramic video.

[0034] In one of the embodiments, determining the interactive highlight of the panoramic video based on the bounding box in the link result includes:

[0035] Determine a first character feature of a character for which interactive highlight recognition is required;

[0036] Determine the character frames contained in the external frame in the link result, and determine the second character feature of the character contained in each character frame;

[0037] Using the first character feature to filter the second character feature, to obtain a third character feature related to the first character feature;

[0038] Based on the bounding box in the link result containing the third character feature, the interactive highlight of the panoramic video is determined.

[0039] In a second aspect, the present disclosure further provides an interactive highlight recognition device. The device comprises:

[0040] A video acquisition module, used to acquire a panoramic video, wherein the panoramic video is composed of multiple panoramic images containing both people and objects;

[0041] An interaction type recognition module is used to recognize the interaction category of a person and an object for each panoramic image frame containing both a person and an object;

[0042] A linking module, configured to link panoramic images with the same interaction category based at least on the interaction category and a preset overlap metric to obtain a linking result;

[0043] An interactive highlight determination module is used to determine the interactive highlights of the panoramic video based on the external frame in the link result.

[0044] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any of the above method embodiments are implemented.

[0045] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0046] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0047] In the above embodiments, by obtaining a panoramic video, and the panoramic video is composed of multiple frames of images containing both people and objects, it is possible to avoid missing key information, which is helpful for subsequent comprehensive and detailed analysis of the entire video scene, and improve the accuracy of interaction highlight recognition. By identifying the interaction category of people and objects for each frame of panoramic image, it is possible to clearly sort out how the people and objects in the video interact with each other, whether it is touching, picking up, putting down or other actions. Clarifying these interaction details can help people better understand the dynamic behavior logic in the scene shown in the video, and further improve the accuracy of interaction recognition. Based on the interaction point, interaction category and preset overlap measurement, the panoramic images indicated by the interaction points with the same interaction category are linked, and the pictures with similar interaction features can be connected in series to ensure temporal coherence. Based on the external frame in the link result, the interaction highlight of the panoramic video is determined, and the most noteworthy and representative interaction part of the video can be extracted, and the most exciting and critical moments of the interaction between people and objects can be extracted. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A diagram showing an application environment of an interactive highlight recognition method in an embodiment;

[0050] Figure 2 is a schematic diagram of a flow chart of an interactive highlight recognition method in one embodiment;

[0051] Figure 3 This is a schematic diagram of the process of step S206 in one embodiment;

[0052] Figure 4 This is a schematic diagram of the process of step S208 in one embodiment;

[0053] Figure 5 Schematic diagram of the process of step S306 in one embodiment;

[0054] Figure 6 A schematic diagram of a flow chart of step S210 in an embodiment;

[0055] Figure 7 Another schematic diagram of a flow chart of step S210 in one embodiment;

[0056] Figure 8 is a schematic block diagram of the structure of an interactive highlight recognition device in one embodiment;

[0057] Fig. 9 is a schematic diagram of the internal structure of a computer device in one embodiment;

[0058] Fig.10 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0061] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0062] The present disclosure provides an interactive highlight recognition method, which can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can obtain the panoramic video stored in the server 104. The panoramic video is composed of multiple frames of panoramic images containing people and objects at the same time. The terminal 102 identifies the interaction category of the person and the object for each frame of the panoramic image containing the person and the object at the same time. The terminal 102 links the panoramic images with the same interaction category based on at least the interaction category and the preset overlap metric to obtain a link result. The terminal 102 determines the interactive highlights of the panoramic video based on the external frame in the link result. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0063] In one embodiment, Figure 2 As shown, an interactive highlight recognition method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0064] S202, obtaining a panoramic video, wherein the panoramic video is composed of multiple panoramic images containing both people and objects.

[0065] Among them, a panoramic image is an image that can show the whole picture of a scene. It does not only present a picture from a certain local angle, but can cover 360 degrees in the horizontal direction, and even cover a larger angle range (such as 180 degrees, etc.) in the vertical direction. Due to the need to identify interactive highlights, each panoramic image not only shows an empty scene, but also includes people and various objects in the scene. For example, in a campus sports meeting scene recorded by a panoramic video, there will be elements such as athletes (people) performing high jump and shot put in the panoramic image.

[0066] Specifically, the terminal can obtain a panoramic video, which needs to be composed of multiple panoramic images containing people and objects at the same time.

[0067] S204, for each panoramic image frame containing both a person and an object, identifying the interaction category of the person and the object.

[0068] Specifically, the trained neural network model can be used to identify the interaction categories of people and objects for each panoramic image frame. The neural network model can be a traditional machine learning classifier model such as a support vector machine (SVM), or a multi-layer fully connected neural network (MLP), or a model with a more complex classification head structure based on CNN or Transformer.

[0069] S206: Based at least on the interaction category and a preset overlap metric, the panoramic images with the same interaction category are linked to obtain a link result.

[0070] Among them, Overlap Metric is a quantitative indicator used to measure the degree of spatial overlap between two or more objects. It is widely used in many fields such as computer vision, image processing, and data analysis.

[0071] Specifically, for each image under each interaction category, two images are selected in turn, and the degree of overlap between them is calculated according to the selected overlap metric (such as the above-mentioned IoU or pixel overlap ratio). For example, the IoU threshold is set to 0.3. If the IoU calculated by the two images is greater than or equal to 0.3, they are considered to have sufficient overlap in the human and object areas and can be linked.

[0072] S208: Determine interactive highlights of the panoramic video based on the bounding box in the link result.

[0073] Specifically, each panoramic image frame contained in the link result is determined, and an external frame in each panoramic image frame is determined. According to the content contained in the external frame, the interactive highlight of the panoramic video is determined.

[0074] In one embodiment, for each panoramic image frame containing both a person and an object, identifying the interaction category of the person and the object includes:

[0075] For each panoramic image frame containing both a person and an object, a person frame of the person and an object frame of the object contained in each panoramic image frame are identified.

[0076] Among them, the character box refers to a bounding box used to mark the area where the character is located in the image. It is usually a rectangle (of course, other shapes can be used according to actual applications, such as irregular polygons to better fit the outline of the character, but rectangles are more commonly used and convenient for calculation and processing). This rectangle tightly surrounds an individual character in the image. Its purpose is to clearly define the specific position range of the character in the entire image. The object box is similar to the character box, but it is a bounding box used to mark the area where various objects other than the character are located in the image. Most of them also use rectangles and other shapes to surround objects. It should be noted that a character box surrounds a character, and an object box surrounds an object.

[0077] Specifically, for each panoramic image containing both people and objects, machine learning methods can be used to identify the person frame and the object frame. For each panoramic image, first, features such as Haar features and HOG (Histogram of Oriented Gradients) features can be extracted. Taking the HOG feature as an example, it characterizes the texture information of the image by calculating the gradient direction histogram of the local area of ​​the image. For a person, its contour edge and other parts will show a specific gradient direction distribution pattern. Objects also have their own unique texture and edge features that can be reflected by HOG. After extracting many features, it is often necessary to use methods such as principal component analysis (PCA) to select and reduce the features, remove some redundant information, and make subsequent classification and recognition more efficient and accurate. For example, PCA can find the most important feature direction in the data, map the high-dimensional feature space to the low-dimensional space while retaining important information as much as possible. Using a large number of sample images with extracted features and marked positions of people and objects (i.e., with corresponding person frames and object frames), we train classifiers such as support vector machines (SVM) and AdaBoost to separate data of different categories (people, different types of objects, etc.), so as to distinguish person samples from other object samples as accurately as possible in the feature space. When facing each new panoramic image, we first extract features and then input the features into the trained classifier. The classifier will determine whether each part of the image belongs to a person or various objects, and then infer possible person frames and object frames based on the corresponding areas. For example, if a certain area is determined to be a person, then a suitable rectangular frame is drawn around the area where the person is located as the person frame. The object frame is similar to the person frame.

[0078] You can also use deep learning to identify person frames and object frames. You can use target detection networks, such as Faster R-CNN (Region-based Convolutional Neural Network), YOLO (You Only LookOnce) series, and SSD (Single Shot MultiBox Detector) networks to identify person frames and object frames.

[0079] In some exemplary embodiments, the input image I∈R H×W Generate features through feature extractor (d is the output step length). Then determine the real human feature points and object feature points in the feature. For example, the real human feature point (x h ,y h ), determine the low-resolution corresponding to the real human feature points Object points are similar. The low-resolution real points are converted into heat maps through Gaussian kernels (the heat map of human points is: Object point heat map T is the number of object categories. The heat map is generated by adding convolution blocks on the feature map, including a 3×3 convolution layer (ReLU activation), a 1×1 convolution layer, and a Sigmoid activation function. By adding four convolution blocks (3×3 convolution layer (ReLU activation) and 1×1 convolution layer) on the feature map, the two-dimensional size and local offset of the human body and object box are estimated respectively, and the size regression loss (such as the loss of height and width) and the local offset loss (such as the loss of the above human feature points) are calculated. in is the set of real human feature points in the training set. The human frame can be adjusted using the loss.

[0080] According to the character frame and the object frame, the interaction points where the character interacts with the object, the interaction categories of the interaction points, and the bounding box containing the character frame and the object frame are determined.

[0081] Among them, an interaction point is usually a key point used to indicate the position of the interaction between a person and an object. The interaction category can be the type of interaction between a person and an object. In different scenes, the interaction type is different. For example, in a basketball scene, the interaction category can include: shoot, pitch, throw, etc. For example, in a volleyball scene, the interaction category can include: dunk, pass, top, etc. Depending on the scene, the interaction category is also different. Those skilled in the art can select a suitable recognition algorithm according to the scene of the panoramic video to determine the interaction category.

[0082] Specifically, the interaction point can be determined by using the geometric center of the character frame and the object frame. The geometric center of the character frame and the object frame can be determined, and the interaction point can be determined in the connecting line segment between the geometric center of the character frame and the object frame. A deep learning model can also be used to predict the interaction point. The model is trained with a large amount of data that is annotated with character-object interaction scenes (including interaction point annotations). The model can predict the location of the interaction point based on the image features in the character frame and the object frame. For example, a convolutional neural network (CNN) is used to extract and predict features of a local image area containing characters and objects, and output the coordinates of the interaction point.

[0083] The step of linking panoramic images with the same interaction category based at least on the interaction category and a preset overlap metric to obtain a link result includes:

[0084] Based on the interaction points, the interaction categories of the interaction points and a preset overlap metric, the panoramic images indicated by the interaction points with the same interaction category are linked to obtain a link result.

[0085] Specifically, for each panoramic image, the interaction points therein must first be identified, and the interaction category of each interaction point must be determined. For example, in an indoor panoramic image, a person (person frame) is identified as holding a cup (object frame), then the position where the hand touches the cup is the interaction point, and the interaction category may be "taking". In addition, it should be noted that since the embodiment of the present disclosure is a panoramic image, there may be multiple people and multiple objects, resulting in multiple interaction points and corresponding interaction categories. According to the specific application scenario, select a suitable overlap metric, such as intersection over union (IoU), Dice coefficient, etc. For interaction points with the same interaction category, calculate their overlap metrics in different panoramic images. According to the overlap metric, the panoramic images indicated by the interaction points with the same interaction category are linked to obtain a link result. It should be noted that not all panoramic images need to be linked.

[0086] In the above interactive highlight recognition method, by obtaining a panoramic video, and the panoramic video is composed of multiple frames of images containing both people and objects, it can avoid missing key information, help to conduct a comprehensive and detailed analysis of the entire video scene in the future, and improve the accuracy of interactive highlight recognition. For each frame of the panoramic image, the character frame of the person and the object frame of the object are identified, and the accurate division and positioning of different subjects in the picture are realized, just like drawing a clear "boundary" for each important element in the video picture, which is convenient for further in-depth analysis around these located elements in the future. According to the character frame and the object frame, the interaction point and the interaction category of the interaction point where the person and the object interact are determined, which can clearly sort out how the person and the object in the video interact with each other, whether it is touching, picking up, putting down or other actions, clarifying these interaction details, can help people better understand the dynamic behavior logic in the scene shown in the video, and further improve the accuracy of interactive recognition. Based on the interaction point, interaction category and preset overlap measurement, the panoramic images indicated by the interaction point with the same interaction category are linked, and the pictures with similar interaction features can be connected in series, thereby ensuring temporal coherence. By determining the interactive highlights of the panoramic video based on the external frame in the link results, we can extract the most noteworthy and representative interactive parts in the video and extract the most exciting and critical moments of the interaction between people and objects.

[0087] In one embodiment, Figure 3 As shown, the determining, based on the character frame and the object frame, the interaction point where the character interacts with the object, the interaction category of the interaction point, and the bounding box containing the character frame and the object frame includes:

[0088] S302, determining a center point of a person in a character frame and a center point of an object in the object frame.

[0089] Specifically, after determining the character frame, the center point of the character frame can be found, and the center point can be the center of the character. The center point of the object frame can also be found, and the center point can be the center point of the object.

[0090] S304: Determine an interaction point based on the character center point and the object center point.

[0091] Specifically, the center point of the character and the center point of the object may be connected, and the interaction point may be determined according to the midpoint between the center point of the character and the center point of the object.

[0092] S306: Based on the character frame and the object frame, generate a circumscribed frame containing the character frame and the object frame.

[0093] Specifically, a bounding box including the character box and the object box may be generated according to the boundary of the character box and the boundary of the object box. Generally, the bounding box may be rectangular.

[0094] S308: Based on the interaction points and the preset interaction categories, generate an interaction heat map that includes the interaction points and matches the interaction categories.

[0095] Among them, the interaction heat map is a data visualization presentation method. The interaction heat map usually maps indicators such as the location, frequency, and intensity of interactive actions on a specific image through visual elements such as color and depth.

[0096] Specifically, the interaction points and a plurality of preset interaction categories may be used to generate a plurality of interaction heat maps that include the interaction points and match the interaction categories.

[0097] In some exemplary embodiments, for example, an interaction point The interaction point can be the midpoint between the center point of the character and the center point of the object, that is, The coordinates of the interaction point at low resolution are Determine the interaction heat map based on the interaction points. The interaction heat map can be: Where K is the preset interaction category.

[0098] S310, calculating the loss of the interaction heat map element by element, and determining the interaction category of the interaction point and the score of the interaction point.

[0099] Among them, the loss is calculated element by element for multiple interaction heat maps matching the interaction category to obtain the loss value, and the interaction category of the interaction point and the score of the interaction point are determined according to the loss value.

[0100] In some exemplary embodiments, an element-by-element focal loss may be applied to the heat map to calculate the point position loss, such as the interaction point heat map loss:

[0101]

[0102] Wherein, α=2, β=4, and N is the number of interaction points in the panoramic image. is the score of category k at point (x, y) in the interaction heat map.

[0103] In this embodiment, by first determining the center point of the character frame and the center point of the object frame, and then determining the interaction point based on these two center points, the positioning of the interaction point is made more scientific and accurate. It helps to analyze the specific interaction behavior between the character and the object in more detail in the subsequent process. Based on the character frame and the object frame, an external frame containing them is generated, which further improves the overall depiction of the relevant elements in the picture and their mutual relationship. The external frame can frame the character and the object as a whole interaction unit, clearly define the scope of the interaction, and facilitate the subsequent analysis of more dimensions around this whole unit. Based on the interaction point and the preset interaction category, a matching interaction heat map is generated, which presents the interaction information in an intuitive and visual way. The interaction heat map is calculated element by element, and then the interaction category and score of the interaction point are determined, so as to realize the quantitative evaluation of the interaction behavior. Through the specific score, the importance, accuracy, etc. of different interaction points can be intuitively compared, so as to accurately determine the current interaction category.

[0104] In one embodiment, Figure 4 As shown, the circumscribed frame includes the interaction point, and based on the interaction point, the interaction category of the interaction point and the preset overlap metric, the panoramic images indicated by the interaction points with the same interaction category are linked to obtain a link result, including:

[0105] S402: According to the scores of the interaction points and the interaction categories of the interaction points, the bounding boxes are sorted and filtered in descending order of the scores to obtain a preset number of target bounding boxes.

[0106] Specifically, since there are interaction points in each bounding box, the bounding boxes with the same interaction category can be grouped according to the interaction category of the interaction point. Then the bounding boxes in each group are sorted again according to the scores of the interaction points, with the ones with larger scores being placed in the front and the ones with smaller scores being placed in the back. The top ranked boxes in each group (for example, the top 10) are taken to obtain a preset number of target bounding boxes.

[0107] S404: Create a corresponding link for each target bounding box in the first panoramic image frame.

[0108] S406, determining each target circumscribed frame that meets a preset connection condition in the panoramic image after the first frame, and sequentially extending the link based on each target circumscribed frame that meets the preset connection condition.

[0109] Among them, each of the target external bounding boxes that meets the preset connection conditions includes: the link is not extended, and the score of the interaction point in the target external bounding box is the highest score, and the average intersection and union ratio between the last target external bounding box in the link and the target external bounding box that meets the preset connection conditions is greater than or equal to the preset overlap metric. Furthermore, each of the target external bounding boxes that meets the preset connection conditions needs to meet the following conditions: 1. It is not selected by other links, 2. It has the highest score, 3. It satisfies ov(Lt)≥tau, tau=0.2, where ov(L, t) is the overlap metric between the last target external bounding box in the current link and the target external bounding box of the current link. It can be understood as the overlap between the last target external bounding box in the current link and the target external bounding box. The overlap between two target external bounding boxes is defined as the average intersection and union ratio (IoU) of the boxes on their overlapping frames. For example, the current link L contains the target bounding box A1, and the target bounding box that meets the preset connection condition is the bounding box A2. Then the bounding box A2 needs to be unselected by other links and have the highest score. At the same time, the average intersection-union ratio between A1 and A2 needs to be greater than 0.2.

[0110] Specifically, in the first frame, a new link is created for each target bounding box. Then, for each interactive category of target bounding boxes in each panoramic image after the first frame, it is determined whether the target bounding box in the first frame meets the preset connection condition. If so, a link is established. In this way, the link is extended.

[0111] In this embodiment, the target external frames that meet the preset linking conditions are linked by linking, which can ensure the accuracy of the final highlight recognition.

[0112] In some embodiments, the method further comprises:

[0113] In response to the presence of the target circumscribed frame that does not meet the preset connection condition in the panoramic image after the first frame, the target circumscribed frame that does not meet the preset connection condition is determined, and a link is created for the target circumscribed frame that does not meet the preset connection condition; the target circumscribed frame in each subsequent frame can also be combined with the target circumscribed frame with the newly created link to determine whether the preset connection condition is met.

[0114] In one embodiment, Figure 5 As shown, the generating of the bounding box containing the character box and the object box based on the character box and the object box includes:

[0115] S502: Based on the character frame and the object frame, generate a bounding rectangle containing the character frame and the object frame, where the bounding rectangle is the bounding box.

[0116] S504: Determine a zoom ratio that matches each frame of the panoramic image based on the size of the bounding box in each frame of the panoramic image.

[0117] S506: adjusting the external frame according to the zoom ratio matching each frame of the panoramic image.

[0118] Specifically, it should be noted that here only one frame of a panoramic image containing one person frame and one object frame is used as an example for explanation. The bounding rectangle containing the person frame and the object frame can be generated according to the person frame and the object frame identified in each frame of the panoramic image. In addition, due to various actions and scenes, the person frame and the object frame may appear in scenes far away. At this time, the bounding box covering the person frame and the object frame will be relatively large. After selecting the highlight area, it may appear that the person and the object in one frame are relatively small, and the person and the object in the next frame are relatively large, resulting in an unsmooth transition. Therefore, according to the size of the bounding box in each frame of the panoramic image, the average value of the bounding box can be selected as the standard bounding box. Then, according to the standard bounding box and the bounding box in each frame, the corresponding scaling ratio of each frame of the panoramic image is determined, and the bounding box is adjusted according to the scaling ratio of each frame of the panoramic image.

[0119] In this embodiment, the method of scaling and adjusting the size of the bounding box of each frame helps to achieve coherent analysis of multiple frames. By making the bounding boxes in different frames comparable and consistent in scale, it is more convenient to track the person-object interaction behavior in consecutive frames.

[0120] In one embodiment, Figure 6 As shown, determining the interactive highlight of the panoramic video based on the external frame in the link result includes:

[0121] S602: Screening is performed based on the interaction categories of the interaction points contained in the bounding boxes in the link results to determine the bounding boxes in the link results that meet a preset target interaction category.

[0122] S604: Determine the interactive highlight of the panoramic video according to the bounding box in the link result that meets the target interaction category.

[0123] Specifically, in some application scenarios, it is necessary to identify fixed target interaction categories. Therefore, the interaction categories can be used for screening, and the bounding boxes in the link results that meet the preset target interaction categories can be retained. The interactive highlights of the panoramic video are determined based on the content in the bounding box in the link results of the preset target interaction categories. For example, in a basketball scene, it is necessary to identify the action of a person shooting, and at this time, only the shooting action needs to be used as the highlight. For other interaction categories, such as dribbling the ball, passing the ball, etc., they are not retained. Therefore, at this time, the shooting interaction category can be directly used as the target interaction category, and only the bounding box containing the shooting interaction category is retained. According to the bounding box containing the shooting interaction category, the interactive highlights of the panoramic video are determined.

[0124] and / or,

[0125] S606: Determine the interactive highlight of the panoramic video based on the average value of the coordinates of the bounding box in the link result.

[0126] Specifically, since the bounding boxes in the link results may not be of the same size, or of the same size but not on the same horizontal line, the identified highlights will fluctuate to a certain extent. Therefore, the coordinates of the bounding boxes in the link results can be determined, and the average value of the bounding boxes can be determined based on the coordinates of the bounding boxes and the number of external frames. The average value of the external boxes can be used to locate the interactive highlights, thereby achieving smoothness.

[0127] In one embodiment, Figure 7 As shown, when there are multiple people in the panoramic image, determining the interactive highlights of the panoramic video based on the external frame in the link result includes:

[0128] S702: Determine a first character feature of a character for which interactive highlight recognition needs to be performed.

[0129] The first character feature may include: appearance-related features, clothing-related features, body-related features, etc. Appearance-related features may include: facial features (such as facial features, facial contours), hairstyle features (such as hairstyle style, hair color). Clothing-related features may include: clothing style, clothing color, etc. Body-related features may include: height, body shape, etc.

[0130] Specifically, a person who needs to perform interactive highlight recognition may be determined first, and then a first person feature may be determined based on the person.

[0131] S704, determining the character frames contained in the circumscribed frame in the link result, and determining the second character feature of the character contained in each character frame.

[0132] Specifically, the character frame contained in the external frame in the link result can be determined, and the character frame usually contains a person. Therefore, the person in the character frame can be identified by feature recognition to determine the second character feature of the person contained in the character frame.

[0133] S706: Filter the second character feature by using the first character feature to obtain a third character feature related to the first character feature.

[0134] Specifically, the link result usually contains multiple bounding boxes, so the similarity between the second character feature and the first character feature of the character in each bounding box can be calculated, and the second character feature with a similarity greater than a similarity threshold can be retained. The retained second character feature can be a third character feature related to the first character feature.

[0135] S708: Determine the interactive highlight of the panoramic video based on the bounding box in the link result containing the third character feature.

[0136] Specifically, after determining the third character feature, a bounding box containing the third character feature may be determined in the link result, and based on the bounding box containing the third character feature, the interactive highlight of the panoramic video may be determined.

[0137] In this embodiment, the first character feature is determined, and the first character feature can be used to quickly lock a specific object of interest from a complex panoramic video scene, thereby improving recognition efficiency and accuracy. The interactive highlights of the panoramic video are determined based on the external frame containing the third character feature, which means that after the previous layers of screening and precise positioning, the determined interactive highlight moments are the wonderful moments that are highly surrounding the target person and have a substantial connection with the target person.

[0138] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0139] Based on the same inventive concept, the disclosed embodiment also provides an interactive highlight recognition device for implementing the interactive highlight recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more interactive highlight recognition device embodiments provided below can refer to the limitations of the interactive highlight recognition method above, and will not be repeated here.

[0140] In one embodiment, Figure 8 As shown, an interactive highlight recognition device 800 is provided, comprising: a video acquisition module 802, an interactive type recognition module 804, a link module 806, and an interactive highlight determination module 808, wherein:

[0141] The video acquisition module 802 is used to acquire a panoramic video, wherein the panoramic video is composed of multiple panoramic images containing people and objects at the same time;

[0142] An interaction type identification module 804 is used to identify the interaction category of the person and the object for each panoramic image frame containing both the person and the object;

[0143] A linking module 806, configured to link panoramic images with the same interaction category based at least on the interaction category and a preset overlap metric to obtain a linking result;

[0144] The interactive highlight determination module 808 is used to determine the interactive highlight of the panoramic video based on the external frame in the link result.

[0145] In one embodiment of the device, the interaction type identification module 804 includes:

[0146] A data recognition module, for identifying, for each panoramic image frame containing both a person and an object, a person frame of the person and an object frame of the object contained in each panoramic image frame;

[0147] An interaction point determination module, used to determine, based on the character frame and the object frame, an interaction point at which the character interacts with the object, an interaction category of the interaction point, and an external frame containing the character frame and the object frame;

[0148] The linking module 806 is further configured to link the panoramic images indicated by the interaction points with the same interaction category based on the interaction points, the interaction categories of the interaction points and a preset overlap metric to obtain a linking result.

[0149] In one embodiment of the device, the interaction point determination module 806 includes:

[0150] The center point determination module determines the center point of the character in the character frame and the center point of the object in the object frame.

[0151] The interaction point determination submodule determines the interaction point based on the character center point and the object center point.

[0152] The circumscribed frame generation module generates a circumscribed frame containing the character frame and the object frame based on the character frame and the object frame.

[0153] An interaction category determination module is used to generate an interaction heat map containing the interaction point and matching the interaction category based on the interaction point and the preset interaction category; calculate the loss of the interaction heat map element by element to determine the interaction category of the interaction point and the score of the interaction point.

[0154] In one embodiment of the device, the external frame includes the interaction point, and the link module 806 includes:

[0155] A sorting module, configured to sort and filter the bounding boxes in descending order of the scores of the interaction points and the interaction categories of the interaction points, to obtain a preset number of target bounding boxes;

[0156] A link creation module, used for creating a corresponding link for each target bounding box in the first frame of the panoramic image;

[0157] A link submodule, configured to determine each target circumscribed frame satisfying a preset connection condition in the panoramic image after the first frame, and sequentially expand the link based on each target circumscribed frame satisfying the preset connection condition;

[0158] Among them, each of the target external bounding boxes that meets the preset connection conditions includes: the link is not expanded, the score of the interaction point in the target external bounding box is the highest score, and the average intersection-and-union ratio between the last target external bounding box in the link and the target external bounding box that meets the preset connection conditions is greater than or equal to a preset overlap metric.

[0159] In one embodiment of the device, the external frame generation module is also used to generate an external rectangle containing the character frame and the object frame based on the character frame and the object frame, and the external rectangle is the external frame; based on the size of the external frame in each frame of the panoramic image, determine the scaling ratio that matches each frame of the panoramic image; and adjust the external frame according to the scaling ratio that matches each frame of the panoramic image.

[0160] In one embodiment of the device, the interactive highlight determination module 808 includes:

[0161] The first determination module is used to screen based on the interaction category of the interaction points contained in the circumscribed frame in the link result, and determine the circumscribed frame in the link result that meets the preset target interaction category; based on the circumscribed frame in the link result that meets the target interaction category, determine the interaction highlight of the panoramic video.

[0162] The second determination module is used to determine the interactive highlight of the panoramic video based on the average value of the coordinates of the circumscribed frame in the link result.

[0163] In one embodiment of the device, the interactive highlight determination module 808 is further used to determine the first character feature of a person who needs to perform interactive highlight recognition; determine the character frame contained in the external frame in the link result, and determine the second character feature of the person contained in each character frame; use the first character feature to filter the second character feature to obtain a third character feature related to the first character feature, and determine the interactive highlight of the panoramic video based on the external frame in the link result containing the third character feature.

[0164] Each module in the above-mentioned interactive highlight recognition device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0165] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store panoramic video data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an interactive highlight recognition method is implemented.

[0166] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an interactive highlight recognition method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse.

[0167] Those skilled in the art will understand that Fig. 9 The structure shown in or 10 is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0169] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0170] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0172] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. An interactive highlight recognition method, characterized in that: The method comprises: Acquire a panoramic video, wherein the panoramic video is composed of multiple panoramic images containing both people and objects; For each panoramic image frame containing both people and objects, identify the interaction category of the people and objects; Based at least on the interaction category and a preset overlap metric, the panoramic images with the same interaction category are linked to obtain a link result; Based on the bounding box in the link result, an interactive highlight of the panoramic video is determined.

2. The method according to claim 1, characterized in that The step of identifying the interaction category of a person and an object for each panoramic image frame containing both a person and an object includes: For each panoramic image frame containing both a person and an object, identifying a person frame of the person and an object frame of the object contained in each panoramic image frame; Determining, according to the character frame and the object frame, an interaction point where the character interacts with the object, an interaction category of the interaction point, and an external frame containing the character frame and the object frame; The step of linking panoramic images with the same interaction category based at least on the interaction category and a preset overlap metric to obtain a link result includes: Based on the interaction points, the interaction categories of the interaction points and a preset overlap metric, the panoramic images indicated by the interaction points with the same interaction category are linked to obtain a link result.

3. The method according to claim 2, characterized in that The step of determining, based on the character frame and the object frame, an interaction point where the character interacts with the object, an interaction category of the interaction point, and a bounding box containing the character frame and the object frame comprises: Determine a center point of a person in a person frame and a center point of an object in an object frame; Determine an interaction point based on the center point of the character and the center point of the object; Based on the character frame and the object frame, generating a bounding box containing the character frame and the object frame; Based on the interaction points and the preset interaction categories, generating an interaction heat map including the interaction points and matching the interaction categories; The loss is calculated element by element on the interaction heat map to determine the interaction category of the interaction point and the score of the interaction point.

4. The method according to claim 3, characterized in that The circumscribed frame includes the interaction point, and based on the interaction point, the interaction category of the interaction point and the preset overlap metric, the panoramic images indicated by the interaction points with the same interaction category are linked to obtain a link result, including: According to the scores of the interaction points and the interaction categories of the interaction points, the bounding boxes are sorted and screened in descending order of the scores to obtain a preset number of target bounding boxes; Creating a corresponding link for each of the target bounding boxes in the first panoramic image frame; Determine each target circumscribed frame that meets a preset connection condition in the panoramic image after the first frame, and sequentially expand the link based on each target circumscribed frame that meets the preset connection condition; Among them, each of the target external bounding boxes that meets the preset connection conditions includes: the link is not expanded, the score of the interaction point in the target external bounding box is the highest score, and the average intersection-and-union ratio between the last target external bounding box in the link and the target external bounding box that meets the preset connection conditions is greater than or equal to a preset overlap metric.

5. The method according to claim 3, characterized in that: The step of generating a bounding box containing the character box and the object box based on the character box and the object box comprises: Based on the character frame and the object frame, generating a bounding rectangle containing the character frame and the object frame, the bounding rectangle being the bounding box; Determine a zoom ratio matching each panoramic image frame based on the size of the bounding box in each panoramic image frame; The external frame is adjusted according to each panoramic image frame matching the zoom ratio.

6. The method according to claim 1, characterized in that The determining the interactive highlight of the panoramic video based on the external frame in the link result includes: Screening is performed based on the interaction categories of the interaction points contained in the bounding box in the link result to determine the bounding box in the link result that meets the preset target interaction category; Determining the interactive highlights of the panoramic video according to the bounding box in the link result that meets the target interaction category; Alternatively, the interactive highlight of the panoramic video is determined based on an average value of the coordinates of the circumscribed frame in the link result.

7. The method according to claim 1, characterized in that The determining the interactive highlight of the panoramic video based on the external frame in the link result includes: Determine a first character feature of a character for which interactive highlight recognition is required; Determine the character frames contained in the external frame in the link result, and determine the second character feature of the character contained in each character frame; Using the first character feature to filter the second character feature, to obtain a third character feature related to the first character feature; Based on the bounding box in the link result containing the third character feature, the interactive highlight of the panoramic video is determined.

8. An interactive highlight recognition device, characterized in that: The device comprises: A video acquisition module, used to acquire a panoramic video, wherein the panoramic video is composed of multiple panoramic images containing both people and objects; An interaction type recognition module is used to recognize the interaction category of a person and an object for each panoramic image frame containing both a person and an object; A linking module, configured to link panoramic images with the same interaction category based at least on the interaction category and a preset overlap metric to obtain a linking result; An interactive highlight determination module is used to determine the interactive highlights of the panoramic video based on the external frame in the link result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images