Method, device and medium for recognizing interactive relationship in video

Abstract

The application relates to a method and device for identifying an interactive relationship in a video, equipment and a medium, wherein the method comprises: performing frame extraction processing on video data to obtain a target picture sequence comprising a target person and an article; generating a relationship topology graph according to the target picture sequence; inputting the relationship topology graph into a preset identification model to enable the preset identification model to identify the relationship topology graph and output an interactive relationship between the target person and the article. The relationship topology graph is established according to the features and relative positions of the pictures in the video, and is input into the preset identification model for identification to obtain an identification result of the interactive relationship, thereby solving the technical problem that the interactive relationship between a person and an article in a video scene cannot be accurately identified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent analysis, and in particular to a method and device for identifying interactive relationships in a video, and a related apparatus and medium. BACKGROUND

[0002] Identifying interactive relationships between characters and objects in a video scene helps to understand the video content. Specifically, technical means and methods are used to analyze the video picture to extract high-level semantic information and help automatically generate label information for downstream tasks. Interactive relationships are the relationships between people and objects, such as a person reading a book or a person riding a bike. Identifying the interactive relationships between characters in a picture helps to promote video content understanding, action recognition, and image description, and other rich commercial service applications.

[0003] Currently, in the related art, the interactive relationship identification technology is mainly based on static images for identification. First, the object type in the picture is detected, and then the interactive relationship between the related objects is predicted. However, the people and objects in a video scene are constantly changing in a short period of time, so it is not possible to accurately identify the interactive relationship between people and objects in a video scene.

[0004] To address the above problems, no effective solutions have been proposed. SUMMARY

[0005] The present application provides a method and device for identifying interactive relationships in a video, and a related apparatus and medium to address the technical problem of being unable to accurately identify the interactive relationship between people and objects in a video scene.

[0006] According to an aspect of an embodiment of the present application, the present application provides a method for identifying interactive relationships in a video, comprising: performing frame extraction processing on video data to obtain a target picture sequence including a target character and an object; generating a relationship topology graph based on the target picture sequence; inputting the relationship topology graph into a preset identification model to enable the preset identification model to identify the relationship topology graph and output the interactive relationship between the target character and the object.

[0007] Optionally, the frame extraction processing on the video data to obtain the target picture sequence including the target character and the object comprises: extracting a frame of picture every interval of a preset time length to obtain an initial picture sequence; detecting whether the initial picture sequence includes the target character and the object; and filtering out pictures in the initial picture sequence that do not include the target character and the object to obtain the target picture sequence.

[0008] Optionally, the generating the relationship topology graph according to the target picture sequence comprises: obtaining a plurality of target pictures in the target picture sequence; generating a frame graph label for each target picture, wherein the frame graph label comprises a person label of a target person or an article label of an article, and the frame graph label of each target picture is different; cropping a target person frame graph feature corresponding to the target person and a target article frame graph feature corresponding to the article in the target picture; determining a topology graph weight of the target person and each article by using the target person frame graph feature and each target article frame graph feature; and generating the relationship topology graph of the target person according to the topology graph weight.

[0009] Optionally, the generating the frame graph label for each target picture comprises: obtaining a time stamp corresponding to the target picture and identifying the target picture to obtain a person name of the target person and an article name of the article; splicing the time stamp and the person name to obtain the person label, and splicing the time stamp and the article name to obtain the article label; and determining the person label and the article label as the frame graph label of the target picture.

[0010] Optionally, the cropping the target person frame graph feature corresponding to the target person and the target article frame graph feature corresponding to the article in the target picture comprises: detecting a person coordinate of the target person and an article coordinate of the article; determining a person contour of the target person by using the person coordinate, and determining an article contour of the article by using the article coordinate; taking a minimum rectangle in which the person contour is located as an initial person frame graph, and taking a minimum rectangle in which the article contour is located as an initial article frame graph; performing feature extraction on the initial person frame graph to obtain the target person frame graph feature, and performing feature extraction on the initial article frame graph to obtain the target article frame graph feature.

[0011] Optionally, the determining the topology graph weight of the target person and each article by using the target person frame graph feature and each target article frame graph feature comprises: determining an intersection region of the target person frame graph feature and the target article frame graph feature, and determining a union region of the target person frame graph feature and the target article frame graph feature; and determining a ratio of the intersection region to the union region as the topology graph weight of the target person and the article.

[0012] Optionally, the generating the relationship topology graph of the target person according to the topology graph weight comprises: initializing an adjacency matrix for the video data; taking the person label as a parent node of the adjacency matrix, and taking a plurality of article labels as child nodes of the adjacency matrix; taking the topology graph weight of the target person and each article as an edge weight of the parent node and each child node to obtain the relationship topology graph.

[0013] According to another aspect of the embodiments of the present application, the present application provides a device for identifying interactive relationship in a video, comprising: a frame extraction module configured to perform frame extraction processing on video data to obtain a target picture sequence comprising a target person and an article; a generation module configured to generate a relationship topology graph corresponding to the interactive relationship between the target person and the article according to a plurality of target pictures in the target picture sequence; and an identification module configured to input the relationship topology graph into a preset identification model to enable the preset identification model to identify the relationship topology graph and output the interactive relationship between the target person and the article.

[0014] According to another aspect of the embodiments of the present application, the present application provides an electronic device comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program executable on the processor, the memory, the processor and the communication interface communicate through the communication bus, and the processor executes the computer program to implement the steps of the above method.

[0015] According to another aspect of the embodiments of the present application, the present application further provides a computer readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the above method.

[0016] The above technical solutions provided by the embodiments of the present application have the following advantages compared with related art:

[0017] The present application provides a method for identifying interactive relationship in a video, comprising: performing frame extraction processing on video data to obtain a target picture sequence comprising a target person and an article; generating a relationship topology graph according to the target picture sequence; and inputting the relationship topology graph into a preset identification model to enable the preset identification model to identify the relationship topology graph and output the interactive relationship between the target person and the article. By establishing a relationship topology graph based on the features and relative positions of pictures in a video and inputting the relationship topology graph into a preset identification model for identification, an interactive relationship identification result is obtained, thereby solving the technical problem that the interactive relationship between a person and an article in a video scene cannot be accurately identified. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the accompanying drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings from these drawings without any creative effort.

[0020] Figure 1 An optional method hardware environment schematic diagram is provided according to the embodiments of the present application.

[0021] Figure 2 An optional video interactive relationship identification method flowchart provided by an embodiment of the present application;

[0022] Figure 3 An optional target picture provided by an embodiment of the present application;

[0023] Figure 4 Another optional target picture provided by an embodiment of the present application;

[0024] Figure 5 An optional video interactive relationship identification device block diagram provided by an embodiment of the present application;

[0025] Figure 6 An optional electronic device structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0027] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only for the convenience of the description of the present application, and have no specific meaning in itself. Therefore, "module" and "component" can be used interchangeably.

[0028] In the related art, the interactive relationship identification technology is mainly based on static image identification, which first needs to detect the object type in the picture, and then predict the interactive relationship between the related objects. However, the people and objects in the video scene are also changing in a short time, so it is impossible to accurately identify the interactive relationship between people and objects in the video scene.

[0029] In order to solve the problems mentioned in the background art, according to an aspect of an embodiment of the present application, an embodiment of a video interactive relationship identification method is provided.

[0030] Optionally, in the embodiments of the present application, the above-mentioned method can be applied to a hardware environment composed of a terminal 101 and a server 103 as shown in Figure 1 Figure 1 ​As shown, the server 103 is connected with the terminal 101 through a network, which can be used to provide services for the terminal or the client installed on the terminal, and a database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The network includes but is not limited to a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer and the like.

[0031] The method in the embodiments of the present application can be executed by the server 103, and can also be executed by the server 103 and the terminal 101 together, as shown in the following. Figure 2 The method can include the following steps:

[0032] Step 201, frame extraction processing is performed on the video data to obtain a target picture sequence including a target person and an object;

[0033] Step 203, a relationship topology graph is generated according to the target picture sequence;

[0034] Step 205, the relationship topology graph is input into a preset recognition model to enable the preset recognition model to recognize the relationship topology graph, and output the interaction relationship between the target person and the object.

[0035] The present application is applied to the technical field of intelligent analysis, and in particular to the recognition of interaction relationship in a video.

[0036] The video includes multiple pictures, and the interaction relationship between the person and the object in the picture is recognized according to the prior art, which cannot accurately express the interaction relationship between the person and the object in the video. Therefore, the present application proposes an interaction relationship recognition and extraction method, which needs to detect the person frame coordinates and the object frame coordinates in the video, extract corresponding features, then establish a relationship graph according to the features and the relative position of the video, and input the graph convolutional neural network model for classification training to obtain the recognition result of the interaction relationship.

[0037] Firstly, the video data is frame-extracted into multiple target pictures containing a target person; then picture features are extracted and a unique frame graph label is generated, and the topology graph weight of the target person and each object in each target picture is calculated; then the number is taken as the node of the topology relationship graph, and the topology graph weight is taken as the edge weight between the nodes to generate a relationship topology graph; finally, the relationship topology graph is input into a preset recognition model for classification to obtain the interaction relationship between the target person and each object.

[0038] It should be noted that the embodiments provided by the present application are specifically described with a single target person, but the present application is applicable to recognizing the interaction relationship between at least one person and each object in the video data, that is, the present application can achieve the interaction relationship between at least one person and each object by recognizing only once.

[0039] As an optional embodiment, the frame extraction processing on the video data to obtain the target picture sequence including the target person and the target object comprises: extracting a picture every interval of a preset time length to obtain an initial picture sequence; detecting whether the initial picture sequence includes the target person and the target object; filtering out the pictures in the initial picture sequence which do not include the target person and the target object to obtain the target picture sequence.

[0040] Optionally, the extracting a picture every interval of a preset time length to obtain an initial picture sequence comprises: obtaining a total time length of the video data, and then evenly dividing the total time length into a plurality of preset time lengths. For example, if the total time length of the video data is 100 seconds, the total time length can be divided into 100 seconds, that is, a picture is extracted every 1 second to obtain an initial picture sequence including 100 pictures.

[0041] Since we need to obtain the interaction relationship between the target person and each target object, we certainly need to be based on the pictures including the target person and the target object. Therefore, the pictures in the initial picture sequence need to be further screened. The pictures in the initial picture sequence are detected whether they include the target person and the target object, and the pictures not including the target person and the target object are deleted from the initial picture sequence to obtain the target picture sequence. The target pictures in the target picture sequence all include the target person.

[0042] The specific steps are: detecting whether the pictures in the initial picture sequence include at least one target person and at least one target object at the same time, deleting the pictures not including the target person or not including the target object from the initial picture sequence to obtain the target picture sequence, and the target pictures in the target picture sequence all include the target person and the target object.

[0043] The pictures are extracted from the video data in time sequence, which can comprehensively express the content of the video data. Then, only the pictures including the target person are selected, and the irrelevant pictures are filtered out. In this way, the error of subsequent recognition can also be reduced to a certain extent.

[0044] As an optional embodiment, the generating a relationship topology graph according to the target picture sequence comprises: obtaining a plurality of target pictures in the target picture sequence; generating a frame label for each target picture, wherein the frame label includes a person label of the target person or an object label of the target object, and the frame label of each target picture is different; cropping a target person frame feature corresponding to the target person and a target object frame feature corresponding to the target object in the target picture; determining a topology graph weight of the target person and each target object by using the target person frame feature and each target object frame feature; and generating a relationship topology graph of the target person according to the topology graph weight.

[0045] Specifically, the frame label generated for each target picture is used to obtain nodes of the relationship topology graph corresponding to the target person and each object, and the target person frame feature corresponding to the target person and the target object frame feature corresponding to the object in the target picture are cropped to determine the relative position of the target person and the object according to the intersection region and the union region of the target person frame feature and the target object frame feature, so as to determine the topology graph weight of the target person and the object, and finally the relationship topology graph is generated by using the topology graph weight between the nodes.

[0046] Next, how to generate the frame label for each target picture is described.

[0047] As an optional embodiment, generating the frame label for each target picture includes: obtaining a timestamp corresponding to the target picture, and identifying the target picture to obtain a person name of the target person and an object name of the object; splicing the timestamp and the person name to obtain a person label, and splicing the timestamp and the object name to obtain an object label; and determining the person label and the object label as the frame label of the target picture.

[0048] Identifying a picture to obtain the object type in the picture is a relatively mature picture processing method, and the picture identification method is not limited in the present application.

[0049] Specifically, the frame label generated for each target picture is composed of a timestamp, a detection type, and a number (a person label or an object label). For example, the timestamp of any target picture is 0000018_00000.760, and it is detected that the target picture includes one person (person) and two cars (car). In order to distinguish different vehicles, the vehicles need to be numbered, for example, car_00 and car_01. The generated frame label includes: 0000018_00000.760_person_00, 0000018_00000.760_car_00, and 0000018_00000.760_car_01.

[0050] As an optional embodiment, cropping the target person frame feature corresponding to the target person and the target object frame feature corresponding to the object in the target picture includes: detecting the person coordinates of the target person and the object coordinates of the object; determining the person contour of the target person by using the person coordinates, and determining the object contour of the object by using the object coordinates; taking the minimum rectangle where the person contour is located as an initial person frame, and taking the minimum rectangle where the object contour is located as an initial object frame; performing feature extraction on the initial person frame to obtain the target person frame feature, and performing feature extraction on the initial object frame to obtain the target object frame feature.

[0051] The target picture is regarded as a plane coordinate system, so that the coordinate area of the target person or object can be determined. The coordinate area of the target person can determine the person contour, and the coordinate area of the object can determine the object contour.

[0052] Figure 3 The target picture provided in the present application is shown in the figure, and the target person and the object are shown in the figure. Figure 3 After the cropping operation, the person contour and the initial person frame diagram, and the object contour and the initial object frame diagram are obtained as shown in Figure 4 The shadow area in the figure is the person contour and the object contour, the rectangular frame outside the person contour is the initial person frame diagram, and the rectangular frame outside the object contour is the initial object frame diagram.

[0053] Optionally, the present application also provides a method for cropping the initial person frame diagram. A straight line parallel to the vertical coordinate axis is drawn through the maximum and minimum horizontal coordinates of the person contour, and a straight line parallel to the horizontal coordinate axis is drawn through the maximum and minimum vertical coordinates of the person contour. The area formed by the four straight lines is the area of the initial person frame diagram, so that the target person frame diagram feature can be determined. The method for determining the target object frame diagram feature is the same as the method for determining the target person frame diagram feature.

[0054] Specifically, all target pictures are subjected to crop processing to be cropped into rectangular initial person frame diagrams and initial object frame diagrams, and then input into a preset residual model for feature extraction. The ImageNet project is a large visual database for visual object recognition software research, and the preset residual model is obtained by training based on ImageNet. The 2048-dimensional features of the target person and the object can be extracted by the preset residual model.

[0055] As an optional embodiment, the determination of the topological graph weight of the target person and the object based on the target person frame diagram feature and the target object frame diagram feature includes: determining the intersection area of the target person frame diagram feature and the target object frame diagram feature, and determining the union area of the target person frame diagram feature and the target object frame diagram feature; and determining the ratio of the intersection area to the union area as the topological graph weight of the target person and the object.

[0056] Specifically, it is considered that the sub-nodes intersected with the person label in the original coordinate frame have an interaction relationship, and the topology graph weight is represented by calculating the IoU (Intersection over Union) value between the target person frame feature and the target object frame feature. IoU is also called overlap or intersection over union. For example, the topology graph weight of the target person and the target object chair is calculated, IoU=(person and chair intersection) / (person and chair union), where the person and chair intersection is the overlapping area of the person frame and the chair frame, and the person and chair union is the sum of the person frame and the chair frame minus the above overlapping area.

[0057] As an optional embodiment, generating the relationship topology graph of the target person according to the topology graph weight comprises: initializing an adjacency matrix for the video data; taking the person feature as the parent node of the adjacency matrix and taking the plurality of object features as the child nodes of the adjacency matrix; and taking the topology graph weight of the target person and each object as the edge weight between the parent node and each child node to obtain the relationship topology graph.

[0058] Specifically, an adjacency matrix is initialized for the video data, and the adjacency matrix represents a matrix of adjacent relationships between vertices, including a plurality of vertices and edges between the vertices.

[0059] The person label of the target person is taken as the parent node, the object label coexisting with the person label in all frame labels is traversed, the object label is taken as the child node, and then the edge weight between the parent node and the child node is determined according to the positional relationship between the target person frame feature and the target object frame feature.

[0060] The IoU value is taken as the weight of the edge connecting the node of the target person and the node of the object, and all child nodes with the person as the parent node are sequentially traversed to obtain the relationship topology graph. It should be noted that the relationship topology graph input into the preset recognition model includes the node feature and the traversed adjacency matrix.

[0061] Finally, the relationship topology graph is input into the preset recognition model, so that the preset recognition model recognizes the relationship topology graph, and outputs the interaction relationship between the target person and the object.

[0062] As an optional embodiment, the training process of the preset recognition model comprises: taking a large number of relationship graphs with the interaction relationship between the person and the article as a training set, taking the labeled interaction relationship as label supervision information, establishing the input (the feature vector of each node and the adjacency matrix representing the relationship between nodes) of the GNN (Graph Neural Networks, graph neural network), and having various operations such as multi-layer graph convolution and activation function, finally obtaining the representation of each node in the graph, and finally classifying the output of the node. For each sample, the cross-entropy loss loss function between the model output and the label is calculated, and then the gradient is back-propagated to the neural network parameters, and the iteration is performed for N generations. After the loss of the classification model is stable, the model capable of recognizing the interaction relationship is obtained. The cross-entropy describes the distance between two probability distributions, and the smaller the cross-entropy, the closer the two are.

[0063] According to another aspect of the embodiments of the present application, as shown in Figure 5 , a device for recognizing interaction relationship in a video is provided, comprising:

[0064] The frame extraction module 502 is configured to perform frame extraction processing on the video data to obtain a target picture sequence comprising a target person and an article.

[0065] The generation module 504 is configured to generate a relationship topology graph according to the target picture sequence.

[0066] The recognition module 506 is configured to input the relationship topology graph into a preset recognition model, so that the preset recognition model recognizes the relationship topology graph, and outputs the interaction relationship between the target person and the article.

[0067] It should be noted that the frame extraction module 502 in this embodiment can be used to execute step 201 in the embodiments of the present application, the generation module 504 in this embodiment can be used to execute step 203 in the embodiments of the present application, and the recognition module 506 in this embodiment can be used to execute step 205 in the embodiments of the present application.

[0068] It should be noted that the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules as part of the device can run in the hardware environment as shown in Figure 1 , and can be implemented by software or hardware.

[0069] Optionally, the frame extraction module 502 is further configured to extract one picture every interval of a preset time length to obtain an initial picture sequence, detect whether the initial picture sequence comprises the target person and the article, and filter out pictures in the initial picture sequence that do not comprise the target person and the article to obtain the target picture sequence.

[0070] Optionally, the generation module 504 comprises:

[0071] an acquisition sub-module, configured to acquire a plurality of target pictures in a target picture sequence;

[0072] a first generation sub-module, configured to generate a frame label for each target picture, wherein the frame label comprises a person label of a target person or an article label of an article, and the frame label of each target picture is different;

[0073] a clipping sub-module, configured to clip out a target person frame feature corresponding to the target person and a target article frame feature corresponding to the article in the target picture;

[0074] a determination sub-module, configured to determine a topological graph weight of the target person and each article by using the target person frame feature and each target article frame feature;

[0075] a second generation sub-module, configured to generate a relationship topological graph of the target person according to the topological graph weight.

[0076] Optionally, the first generation sub-module is further configured to acquire a timestamp corresponding to the target picture, and identify the target picture to obtain a person name of the target person and an article name of the article; splice the timestamp and the person name to obtain the person label, and splice the timestamp and the article name to obtain the article label; and determine the person label and the article label as the frame label in the target picture.

[0077] Optionally, the clipping sub-module is further configured to detect a person coordinate of the target person and an article coordinate of the article; determine a person contour of the target person by using the person coordinate, and determine an article contour of the article by using the article coordinate; take a minimum rectangle in which the person contour is located as an initial person frame, and take a minimum rectangle in which the article contour is located as an initial article frame; perform feature extraction on the initial person frame to obtain the target person frame feature, and perform feature extraction on the initial article frame to obtain the target article frame feature.

[0078] Optionally, the determination sub-module is further configured to determine an intersection region of the target person frame feature and the target article frame feature, and determine a union region of the target person frame feature and the target article frame feature; and determine a ratio of the intersection region to the union region as the topological graph weight of the target person and the article.

[0079] Optionally, the second generation sub-module is further configured to initialize an adjacency matrix for video data; take the person label as a parent node of the adjacency matrix, and take a plurality of article labels as child nodes of the adjacency matrix; and take the topological graph weight of the target person and each article as an edge weight of the parent node and each child node to obtain the relationship topological graph.

[0080] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such asFigure 6 As shown, the electronic device includes a memory 601, a processor 603, a communication interface 605, and a communication bus 607. The memory 601 stores a computer program executable on the processor 603. The memory 601 and the processor 603 communicate through the communication interface 605 and the communication bus 607. The processor 603 executes the computer program to implement the steps of the above method.

[0081] The memory and the processor in the electronic device communicate through the communication bus and the communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0082] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0083] The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. The processor can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0084] According to another aspect of the embodiments of the present application, a computer readable medium having non-volatile program code executable by a processor is also provided.

[0085] Optionally, in the embodiments of the present application, the computer readable medium is configured to store program code for the processor to execute the following steps:

[0086] In step 201, frame extraction processing is performed on the video data to obtain a target picture sequence including a target person and an object.

[0087] In step 203, the relationship topology graph is generated according to the target picture sequence.

[0088] In step 205, the relationship topology graph is input into a preset recognition model, so that the preset recognition model recognizes the relationship topology graph, and outputs the interaction relationship between the target person and the article.

[0089] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be described here again.

[0090] When the embodiments of the present application are implemented, the above-mentioned embodiments can be referred to, and have corresponding technical effects.

[0091] It can be understood that the embodiments described herein can be realized by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0092] For software implementation, the technologies described herein can be realized by units performing the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0093] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-mentioned system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here again.

[0095] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. The division of the modules is merely logical function division. There can be another division manner for the actual implementation, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0096] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0097] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0098] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or the parts of the technical solutions that make contributions to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, and various other media that can store program codes. It should be noted that, in this document, relational terms such as "first" and "second" and the like are used only to distinguish one entity or action from another entity or action, and do not necessarily require or imply that there is any such actual relationship or order between these entities or actions. In addition, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a list of elements does not exclude other elements not explicitly listed, or other elements inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0099] The foregoing detailed description of the application has been presented for purposes of illustration and description. Various modifications and changes can be made to these embodiments without departing from the spirit and scope of the application. It is intended that the scope of the application should not be limited by the particular representative embodiments described above. Rather, the general scope of the application is to be determined by the appended claims and their equivalents.

Claims

1. A method for recognizing an interactive relationship in a video, characterized by, The method comprises: frame extraction processing is performed on video data to obtain a target picture sequence comprising a target person and an article; a relationship topology graph is generated according to the target picture sequence; the relationship topology graph is input into a preset identification model, so that the preset identification model identifies the relationship topology graph, and outputs an interaction relationship between the target person and the article; the relationship topology graph is generated according to the topology weight, which comprises: obtaining a plurality of target pictures in the target picture sequence; a frame graph label is generated for each target picture, wherein the frame graph label comprises a person label of the target person and an article label of the article, and the frame graph label of each target picture is different; a target person frame graph feature corresponding to the target person and a target article frame graph feature corresponding to the article in the target picture are cropped out; the topology graph weight of the target person and each article is determined by using the target person frame graph feature and each target article frame graph feature; and the relationship topology graph of the target person is generated according to the topology graph weight. the relationship topology graph is generated according to the topology graph weight, which comprises: an adjacency matrix is initialized for the video data; the person label is taken as a parent node of the adjacency matrix, and a plurality of article labels are taken as child nodes of the adjacency matrix; and the topology graph weight of the target person and each article is taken as an edge weight of the parent node and each child node, so as to obtain the relationship topology graph.

2. The method of claim 1, wherein, The frame extraction processing on the video data to obtain the target picture sequence comprising the target person and the article comprises: extracting a picture every interval of a preset time length to obtain an initial picture sequence; detecting whether the initial picture sequence comprises the target person and the article; filtering out the pictures in the initial picture sequence that do not comprise the target person and the article to obtain the target picture sequence.

3. The method of claim 1, wherein, The frame graph label is generated for each target picture, which comprises: acquiring a timestamp corresponding to the target picture and identifying the target picture to obtain a person name of the target person and an article name of the article; splicing the timestamp and the person name to obtain the person label, and splicing the timestamp and the article name to obtain the article label; determining the person label and the article label as the frame graph label of the target picture.

4. The method of claim 1, wherein, The target person frame graph feature corresponding to the target person and the target article frame graph feature corresponding to the article in the target picture are cropped out, which comprises: detecting person coordinates of the target person and article coordinates of the article; determining a person contour of the target person by using the person coordinates, and determining an article contour of the article by using the article coordinates; taking a minimum rectangle in which the person contour is located as an initial person frame graph, and taking a minimum rectangle in which the article contour is located as an initial article frame graph; performing feature extraction on the initial person frame graph to obtain the target person frame graph feature, and performing feature extraction on the initial article frame graph to obtain the target article frame graph feature.

5. The method of claim 1, wherein, The determining the topological graph weight of the target person and each of the objects by using the target person frame graph feature and each of the target object frame graph features comprises: determining an intersection region of the target person frame graph feature and the target object frame graph feature, and determining a union region of the target person frame graph feature and the target object frame graph feature; determining a ratio of the intersection region to the union region as the topological graph weight of the target person and the object.

6. An apparatus for recognizing an interactive relationship in a video, characterized by comprising: Comprise: a frame extraction module configured to perform frame extraction processing on video data to obtain a target picture sequence comprising a target person and an object; a generation module configured to generate a relationship topological graph according to the target picture sequence; an identification module configured to input the relationship topological graph into a preset identification model, so that the preset identification model identifies the relationship topological graph, and outputs an interaction relationship between the target person and the object; The generation module is further configured to obtain a plurality of target pictures in the target picture sequence; generate a frame graph label for each of the target pictures, wherein the frame graph label comprises a person label of the target person and an object label of the object, and the frame graph label of each of the target pictures is different; crop a target person frame graph feature corresponding to the target person and a target object frame graph feature corresponding to the object in the target picture; determine the topological graph weight of the target person and each of the objects by using the target person frame graph feature and each of the target object frame graph features; and generate the relationship topological graph of the target person according to the topological graph weight. The generation module is further configured to initialize an adjacency matrix for the video data; take the person label as a parent node of the adjacency matrix, and take a plurality of object labels as child nodes of the adjacency matrix; take the topological graph weight of the target person and each of the objects as an edge weight of the parent node and each of the child nodes to obtain the relationship topological graph.

7. An electronic device comprising a memory, a processor, a communication interface and a communication bus, the memory storing a computer program executable on the processor, the memory, the processor communicating through the communication bus and the communication interface, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 5.

8. A computer readable medium having a non-transitory program code executable by a processor, the program code comprising instructions for: The program code causes the processor to execute the method of any one of claims 1 to 5. The program code causes the processor to execute the method of any one of claims 1 to 5.

Images