Picture intelligent identification method and device applied to video communication

By analyzing the node graph structure and posture vectors in the video data sample, and calculating the behavioral similarity and contribution, the posture recognition problem caused by different user action habits is solved, the recognition stability and adaptability in video communication is improved, and the user is enhanced Interactive experience.

CN119068390AActive Publication Date: 2024-12-03STATE GRID INFORMATION & TELECOMM BRANCH
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411094032.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

During the video communication process, the existing intelligent image recognition methods have poor stability, accuracy and adaptability of posture recognition, and cannot effectively identify the user's target behavior.

Method used

By obtaining video data samples, extracting the node graph structure and posture vectors of different poses under different behaviors, analyzing common and unique features, calculating the similarity and node contribution between behaviors, generating similar score values ​​to identify behavior types and output dynamics Special effects.

Benefits of technology

It improves the stability, accuracy and adaptability of different behaviors, and enhances the user's sense of interaction and participation in the video communication process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068390A_ABST
    Figure CN119068390A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image communication, and provides an intelligent picture recognition method and device applied to video communication, and the method comprises the steps: obtaining a video data sample, carrying out the feature extraction of each piece of video frame data in the video data sample, obtaining a node graph structure and a posture vector of different postures under different behaviors, and carrying out the feature extraction; and then obtaining common features and specific features of different postures under different behaviors, and improving the recognition adaptability to one behavior type by combining the common features and the specific features. And meanwhile, based on the existing behavior type characteristics, analyzing the included body node contribution degree and similarity to obtain a behavior type corresponding to the current behavior in the current video frame, and outputting a dynamic special effect corresponding to the behavior type at a corresponding position on a user screen. In this way, the recognition stability, accuracy and adaptability of different behaviors can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image communication, and in particular to a method and device for intelligently identifying pictures applied to video communication. Background Art

[0002] With the rapid development of 5G networks and video communication technology, video calls and live video broadcasts have become increasingly inseparable from people's lives. The use of video communication technology can fully narrow the distance between users and enrich people's daily lives. At the same time, with the gradual increase in users' demand for interactivity and fun in video communication, users are no longer satisfied with simple video communication and live video broadcasts. Therefore, by using intelligent image recognition technology, the user's actions, expressions and other information during video communication can be recognized and corresponding dynamic special effects or animations can be added, thereby achieving the purpose of enhancing the user's sense of participation and interaction during video communication.

[0003] Existing intelligent image recognition methods usually perform facial recognition, expression tracking, gesture recognition and other operations on the characters in the video, and then dynamically add virtual decorations and special effects according to the postures shown by the users, so as to enhance the user's sense of participation and interactivity. However, in actual application, some target behaviors can be achieved through a variety of different action postures. Due to the different action habits and physical conditions of different users, their postures when obtaining dynamic special effects during video calls or live broadcasts will also be different. As a result, the postures shown by the users do not match the target behaviors stored in the system, resulting in the inability to recognize the postures, resulting in poor stability, accuracy and adaptability of system recognition. Summary of the invention

[0004] In view of the above problems, the present application provides a method and device for intelligent image recognition applied to video communication, which can improve the recognition stability, accuracy and adaptability of different behaviors.

[0005] In a first aspect, an embodiment of the present application provides a method for intelligently identifying pictures applied to video communication, including:

[0006] Get video data samples;

[0007] Extract features from each video frame in the video data sample to obtain the node graph structure and posture vector of different postures under different behaviors;

[0008] Perform feature analysis on each video frame data in the video data sample according to the node graph structure and posture vector, and obtain the common features and unique features of different postures under different behaviors;

[0009] According to the common features and unique features of the current behavior and different postures under different behaviors in the current video frame, obtain the first similar video frame data in the video data sample corresponding to the current video frame;

[0010] Analyze the contribution of each node in the node graph structure of the current behavior in the current video frame to obtain the recognition contribution of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data;

[0011] According to the recognition contribution degree of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data, a similarity score value of the current behavior in the current video frame to the posture under the behavior corresponding to the first similar video frame data is obtained;

[0012] According to the similarity score value, the behavior type corresponding to the current behavior in the current video frame is obtained, and the dynamic special effects corresponding to the behavior type are output at the corresponding position on the user's screen.

[0013] In a possible implementation, feature extraction is performed on each video frame data in the video data sample to obtain node graph structures and posture vectors of different postures under different behaviors, including:

[0014] Use human body detection algorithm to locate the user position in each frame of the video data sample;

[0015] According to the user position in each frame of the video data sample, the posture estimation algorithm is used to identify the key nodes of the human body in each frame, and the node graph structure of different postures under different behaviors is obtained;

[0016] The key nodes in the node graph structure are vectorized to obtain posture vectors of different postures under different behaviors.

[0017] In a possible implementation, feature analysis is performed on each video frame data in the video data sample according to the node graph structure and the posture vector to obtain common features and unique features of different postures under different behaviors, including:

[0018] The maximum common subgraph of different postures under each same behavior is obtained using the maximum common subgraph acquisition algorithm to obtain the common features of different postures under each same behavior;

[0019] According to the common features of different postures under each same behavior, the corresponding node graph structure is deleted to obtain the unique features of different postures under each same behavior.

[0020] In a possible implementation, it includes:

[0021] The cosine similarity calculation is performed on the posture vectors corresponding to the common features of different postures under any two different behaviors to obtain the first similarity score of the common features between any two behaviors;

[0022] The maximum common subgraph between two behaviors whose first similarity scores are greater than a first preset threshold is used to obtain the maximum common subgraph between the two behaviors using a maximum common subgraph acquisition algorithm, and the common features are corrected according to the maximum common subgraph to obtain the common features of different postures under different behaviors.

[0023] In a possible implementation, obtaining first similar video frame data in a video data sample corresponding to the current video frame according to common features and unique features of different postures under different behaviors in the current video frame includes:

[0024] Compare the current behavior in the current video frame with common features of different behaviors to obtain multiple first behaviors corresponding to the current behavior in the current video frame;

[0025] The current behavior in the current video frame is compared with the unique features of different postures under the multiple first behaviors to obtain the first similar video frame data in the video data sample corresponding to the current video frame.

[0026] In a possible implementation, a contribution degree analysis is performed on each node in the node graph structure of the current behavior in the current video frame to obtain the recognition contribution degree of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data, wherein the calculation formula of the recognition contribution degree is:

[0027]

[0028] Among them, G v represents the recognition contribution of node v in the current behavior j to node v in other behaviors r, f j,v represents the frequency of occurrence of node v in the current behavior j; represents the mean support of all connection modes of node v in the current behavior j; f r,v represents the frequency of occurrence of node v in other behaviors r; represents the mean support of all connection modes of node v in other behaviors r; exp represents an exponential function with base e.

[0029] In a possible implementation, according to the recognition contribution of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data, a similarity score value of the current behavior in the current video frame to the posture under the behavior corresponding to the first similar video frame data is obtained, wherein the calculation formula of the similarity score value is:

[0030]

[0031] Among them, P i Indicates the behavior in the current video frame for the posture k under behavior i iThe similarity score value of n and m represents the number of nodes of behavioral features and posture features respectively; (xs iv ×G iv ) represents the matching degree between node v and behavior i; (xs kiv +G kiv ) represents the node v and posture k i The degree of matching; xs iv and xs kiv Represents the node v and behavior i or posture k in the current video frame respectively i Similarity of the next node v; G iv and G kiv Respectively represent the behavior i or posture k of node v in the current video frame i The recognition contribution of the next node v; Norm represents the normalization processing function.

[0032] In a possible implementation, the behavior type corresponding to the current behavior in the current video frame is obtained according to the similarity score value, and a dynamic special effect corresponding to the behavior type is output at a corresponding position on the user screen, including:

[0033] The similarity score value of the posture of the current behavior in the current video frame to the corresponding behavior of the first similar video frame data is compared with the second preset threshold, and whether to output dynamic special effects is determined according to the comparison result.

[0034] In a possible implementation, it includes:

[0035] When the similarity score value is greater than or equal to a second preset threshold, a dynamic special effect corresponding to the behavior type is output at a corresponding position on the user screen;

[0036] When the similarity score value is less than the second preset threshold, no dynamic special effects are output on the user screen.

[0037] In a second aspect, an embodiment of the present application provides an intelligent image recognition device for video communication, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the various possible embodiments of the first aspect.

[0038] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute various possible implementation embodiments of the first aspect.

[0039] The beneficial effect of the present application is that the present application can obtain the common features and unique features of the posture under each behavior type through in-depth analysis of each behavior type, and improve the recognition adaptability of a behavior type by combining the common features and unique features. At the same time, based on the existing behavior type characteristics, the contribution degree and similarity of the body nodes included in it are analyzed, that is, different behaviors have different focuses on node analysis, which improves the recognition stability, accuracy and adaptability of different behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of a method for intelligently identifying pictures applied to video communication provided by an embodiment of the present application;

[0041] Figure 2 It is a structural schematic diagram of the largest common subgraph in two node graph structures provided in an embodiment of the present application;

[0042] Figure 3 It is a structural block diagram of a picture intelligent recognition device applied to video communication provided in an embodiment of the present application;

[0043] Figure 4 It is a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0045] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0046] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0047] When making a video call or live video broadcast, when the user wants to obtain the required dynamic special effects, he usually performs his habitual movements. Since some target behaviors can usually be achieved through a variety of different action postures, and different users have different action habits, their postures when obtaining dynamic special effects will also be different during video calls or live broadcasts. At this time, the posture performed by the user may not match the target behavior stored by the system, resulting in unrecognition, resulting in poor recognition stability, accuracy and adaptability of the image intelligent recognition system.

[0048] This application analyzes different posture performances of a certain behavior, obtains the common node spatial distribution characteristics and unique node spatial distribution characteristics between different postures, assigns different behavior recognition contribution values ​​to different nodes, and finally calculates the similarity values ​​of different behaviors, thereby improving the ability to recognize user postures in pictures during video communication.

[0049] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0050] See also Figure 1 As shown, the embodiment of the present application discloses a method for intelligent image recognition applied to video communication, including:

[0051] Step S11, obtaining a video data sample;

[0052] Step S12, extracting features from each video frame data in the video data sample to obtain node graph structures and posture vectors of different postures under different behaviors;

[0053] Step S13, performing feature analysis on each video frame data in the video data sample according to the node graph structure and the posture vector, to obtain common features and unique features of different postures under different behaviors;

[0054] Step S14, obtaining first similar video frame data in the video data sample corresponding to the current video frame according to the common features and unique features of the current behavior and different postures under different behaviors in the current video frame;

[0055] Step S15, analyzing the contribution of each node in the node graph structure of the current behavior in the current video frame to obtain the recognition contribution of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data;

[0056] Step S16, obtaining a similarity score value of the current behavior in the current video frame to the posture in the behavior corresponding to the first similar video frame data according to the recognition contribution degree of each node of the current behavior to the posture in the behavior corresponding to the first similar video frame data;

[0057] Step S17, obtaining the behavior type corresponding to the current behavior in the current video frame according to the similarity score value, and outputting the dynamic special effects corresponding to the behavior type at the corresponding position on the user's screen.

[0058] The required video data is collected through multiple channels, that is, the corresponding triggering behavior videos of preset dynamic special effects or animations. The collected video objects include relevant posture videos of different ages, skin colors, and environments, so that the collected video data samples are as perfect as possible. Specifically, determine the type of behavior to be collected, such as scissors hand, heart, like, etc.; according to the determined behavior collection type, collect relevant video data samples through multiple channels and sources; pre-process the collected video data samples, that is, noise reduction, enhancement, etc.; classify the different posture video data samples corresponding to the same behavior and mark them as X 11 (i.e., behavior 1 posture 1), and so on for other video data samples, such as X 21 , X 22 (i.e., behavior 2 posture 1, behavior 2 posture 2), the processed video data samples belonging to the same behavior are stored in a database, and a video data sample set of all different behaviors is constructed to complete the collection of the required data samples.

[0059] In the above-mentioned embodiment steps, firstly, a video data sample is obtained, and feature extraction is performed on each video frame data in the video data sample to obtain the node graph structure and posture vector of different postures under different behaviors, and then the common features and unique features of different postures under different behaviors are obtained, and then the recognition contribution degree of each node of the current behavior to the posture under the corresponding behavior of the first similar video frame data and the similarity score value of the posture under the corresponding behavior of the first similar video frame data in the current behavior in the current video frame are obtained, and finally, the behavior type corresponding to the current behavior in the current video frame is obtained according to the common features, unique features, recognition contribution degree and similarity score value, and the dynamic special effects corresponding to the behavior type are output at the corresponding position on the user's screen. The present application can obtain the common features and unique features of the posture under each behavior type by deeply analyzing each behavior type, and improve the recognition adaptability for a behavior type by combining the common features and unique features. At the same time, based on the existing behavior type features, the contribution degree and similarity of the body nodes included therein are analyzed, that is, different behaviors have different focuses on node analysis, and the recognition stability, accuracy and adaptability for different behaviors are improved.

[0060] It should be noted that in the subsequent description process, behavior refers to the triggering condition of a certain special effect. For example, making a heart shape with hands is a behavior, and posture refers to the way of performing this behavior, such as making a heart shape with fingers, making a heart shape with hands, making a heart shape with arms, etc.

[0061] Since some behaviors can be achieved through a variety of different postures, this means that different postures under the same behavior have certain similarities, which are mainly reflected in the similarity of the spatial structure of the human body nodes. However, there are also differences in the spatial structure of the nodes between different postures, so different postures can be distinguished. In order to obtain the common features and unique features between different postures under the same behavior, it is necessary to obtain the node graph structure of each sample, determine the similarities and differences between different postures through comparison, and finally extract different posture features for different behaviors.

[0062] In an optional embodiment of the present application, feature extraction is performed on each video frame data in the video data sample to obtain node graph structures and posture vectors of different postures under different behaviors, including:

[0063] Use human body detection algorithm to locate the user position in each frame of the video data sample;

[0064] According to the user position in each frame of the video data sample, the posture estimation algorithm is used to identify the key nodes of the human body in each frame, and the node graph structure of different postures under different behaviors is obtained;

[0065] The key nodes in the node graph structure are vectorized to obtain posture vectors of different postures under different behaviors.

[0066] It should be noted that the node graph structure and posture feature vector can reflect the specific behavioral characteristics of the user in the picture. First, the human posture features in each video data sample are extracted using the existing algorithm to obtain the feature vectors and node graph structures of different postures under different behaviors.

[0067] Specifically, firstly, a human body detection algorithm (such as Haar Cascades Classifier, HOG+SVM algorithm, etc.) is used to locate the user position in each frame of the video data sample; according to the known user body position in the video frame, a posture estimation algorithm (such as OpenPose, AlphaPose algorithm, etc.) is used to identify the key nodes of the human body in each frame, and the key nodes are connected in combination with the human body structure to obtain a node graph structure; the user key point features (node ​​graph structure) obtained above are vectorized to obtain the posture vector of each video data sample in the current behavior type; the above steps are repeated to obtain the posture vectors of the video data samples of all postures under different behaviors to complete the feature extraction.

[0068] It should be noted that the length of each posture vector is the same (including all human body nodes) and the meaning of the corresponding dimensions is the same, and each element thereof corresponds to the position or angle information of a key node in the image.

[0069] In an optional embodiment of the present application, feature analysis is performed on each video frame data in the video data sample according to the node graph structure and the posture vector to obtain common features and unique features of different postures under different behaviors, including:

[0070] The maximum common subgraph of different postures under each same behavior is obtained using the maximum common subgraph acquisition algorithm to obtain the common features of different postures under each same behavior;

[0071] According to the common features of different postures under each same behavior, the corresponding node graph structure is deleted to obtain the unique features of different postures under each same behavior.

[0072] It should be noted that the maximum common subgraph refers to the largest subgraph that can be found in two or more graph structures. This subgraph exists in all compared graphs at the same time. For two input graphs with different nodes (called graph pairs), the maximum common subgraph (such as the attached Figure 2 As shown), and this subgraph is required to be connected and isomorphic, such as Figure 2 Therefore, the node graph structure of different postures under the same behavior must have a maximum common subgraph, which is the common feature of the current behavior, and the node information outside the maximum common subgraph belongs to the unique features of each posture.

[0073] During video communication, the largest common subgraph of a certain behavior is searched for in the video frame image to identify the possible behavior type of the image, and then the search is performed based on the unique features of different postures under each behavior type, thereby realizing intelligent recognition of the video frame image.

[0074] Specifically, first, the existing maximum common subgraph acquisition algorithm (MCSA) is used to obtain the maximum common subgraph under behavior i, denoted as GT i , the node graph structure is the common feature of behavior i; then, the unique features of any posture under behavior i are extracted, and the maximum common subgraph GT is used i All node graph structures under posture k are deleted respectively to obtain the unique node graph structure of all samples under posture k; repeat the above steps to obtain the common features and unique features of different postures under different behaviors, and vectorize all the common features and unique features.

[0075] It should be noted that in order to ensure the accuracy of user behavior recognition, that is, the uniqueness of the common features of each behavior, it is also necessary to analyze the similarity of the common features between different behaviors. When the similarity of the common features between different behaviors is high, it is necessary to analyze and delete the largest common subgraph of the common features of the behavior type to ensure stability during image recognition.

[0076] In an optional embodiment of the present application, it includes:

[0077] The cosine similarity calculation is performed on the posture vectors corresponding to the common features of different postures under any two different behaviors to obtain the first similarity score of the common features between any two behaviors;

[0078] The maximum common subgraph between two behaviors whose first similarity scores are greater than a first preset threshold is used to obtain the maximum common subgraph between the two behaviors using a maximum common subgraph acquisition algorithm, and the common features are corrected according to the maximum common subgraph to obtain the common features of different postures under different behaviors.

[0079] Specifically, in the embodiment of the present application, firstly, the similarity scores μ of the common feature vectors of behavior i and all other behaviors are calculated respectively. u (cosine similarity), that is, the vector similarity score with the u-th behavior; set a threshold to screen the similarity score. When the vector similarity ε between any two behaviors is greater than 0.5 (the first preset threshold), it is considered that the common features of the two behaviors are too similar; the maximum common subgraph of the two behaviors is obtained again, and the subgraphs of the two behaviors are corrected using the obtained maximum common subgraph to obtain the corrected node graph structure GT′ of behavior i i , which is the common feature of the final behavior i.

[0080] It should be further explained that the correction method here is to compare and screen the re-acquired common subgraph with the characteristic subgraph of the original behavior to obtain the behavior characteristic subgraph without the common subgraph, which is the common feature of this behavior.

[0081] In an optional embodiment of the present application, obtaining first similar video frame data in a video data sample corresponding to the current video frame according to common features and unique features of different postures under different behaviors in the current video frame includes:

[0082] Compare the current behavior in the current video frame with common features of different behaviors to obtain multiple first behaviors corresponding to the current behavior in the current video frame;

[0083] The current behavior in the current video frame is compared with the unique features of different postures under the multiple first behaviors to obtain the first similar video frame data in the video data sample corresponding to the current video frame.

[0084] It should be noted that in the process of intelligent image recognition, we first determine the behavior type (such as scissors hands, heart, likes, etc.) that the video frame may belong to by comparing the common features (node ​​graph structure) of the video frame with different behaviors, and then determine the posture to which the video frame belongs by comparing the unique features of different postures under the behavior type.

[0085] However, when comparing the video frame with the feature node graph structure, different behaviors and different postures under different behavior types must contain the same nodes in their node graph structures, and there may still be local similarities. Therefore, in order to improve the adaptability of intelligent image recognition, the recognition contribution degree of different nodes in the node graph structure is analyzed so that special nodes have higher weights in the image recognition process.

[0086] In an optional embodiment of the present application, a contribution degree analysis is performed on each node in the node graph structure of the current behavior in the current video frame to obtain the recognition contribution degree of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data, wherein the calculation formula for the recognition contribution degree is:

[0087]

[0088] Among them, G v represents the recognition contribution of node v in the current behavior j to node v in other behaviors r, f j,v represents the frequency of occurrence of node v in the current behavior j; represents the mean support of all connection modes of node v in the current behavior j; f r,v represents the frequency of occurrence of node v in other behaviors r; represents the mean support of all connection modes of node v in other behaviors r; exp represents an exponential function with base e.

[0089] Specifically, in the embodiment of the present application, the node graph structures of different behaviors are first compared and analyzed to obtain the occurrence frequency f of node v in a certain behavior or posture feature. v (Number of node graph structures / Number of node graph structures with node v). The lower the frequency of node v, the higher the contribution. The lower the frequency of node v in other behaviors, the higher the frequency of node v in the current behavior, the higher the contribution. This means that the more unique node v is in the current behavior, the higher the contribution. Then, the connection relationship between nodes is analyzed, and the frequent item sets connected to node v are obtained through data mining algorithms (FP-Growth algorithm, etc.). The overall support ZC of the frequent item set r corresponding to node v is v,rThe lower it is, the higher the contribution of node v. The definition of frequent itemsets is unique to the FP-Growth algorithm, and the calculation of support is similar. They all belong to existing algorithms and definitions, indicating the frequency of occurrence of different connection methods in the node connection process. The lower the support of a node's frequent itemsets in all behaviors, the higher the support in the current behavior, and the higher the contribution.

[0090] It should be further explained that when obtaining the contribution of a node by analyzing the support of the connection mode, it is necessary to analyze all frequent item sets corresponding to a node and calculate their mean as the contribution influencing parameter (support mean), that is,

[0091] Through the calculation formula of the recognition contribution degree, each human body node can be traversed according to the node graph structure of different postures under different behaviors to obtain the recognition contribution degree of each node for different behaviors and different postures.

[0092] In an optional embodiment of the present application, according to the recognition contribution of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data, a similarity score value of the current behavior in the current video frame to the posture under the behavior corresponding to the first similar video frame data is obtained, wherein the calculation formula of the similarity score value is:

[0093]

[0094] Among them, P i Indicates the behavior in the current video frame for the posture k under behavior i i The similarity score value of n and m represents the number of nodes of behavioral features and posture features respectively; (xs iv ×G iv ) represents the matching degree between node v and behavior i; Represents node v and posture k i The degree of matching; xs iv and Represents the node v and behavior i or posture k in the current video frame respectively i Similarity of the next node v; G iv and Respectively represent the behavior i or posture k of node v in the current video frame i The recognition contribution of the next node v; Norm represents the normalization processing function.

[0095] It should be noted that a comparative analysis is performed on the user's behaviors and postures in the video frames, and the similarity scores between the node graph structure in the video frames and the feature node graph structure are calculated. During the comparison process, corrections are made based on the contribution of the nodes to different behaviors and postures (the correction process is to calculate the similarity of different postures under the current behavior when calculating the behavior similarity), so as to obtain the similarity score values ​​between the user in the video frame and different behaviors.

[0096] Specifically, in the embodiment of the present application, first, the sampling interval of the video frame of the video communication is 2s (which can be set according to the specific situation, such as key frame analysis, etc.), the current video frame A is obtained, and the node graph structure corresponding to the video frame A is obtained; then, the node graph structure of the video frame A is compared with the behavior and posture characteristics, and the similarity xs between the node v connection mode of the behavior and posture characteristics is obtained. iv 、xs kiv ,Similarity is to compare the differences in the node graph structure (features), such as the differences in node connection methods, the differences in the nodes contained, and the differences in the spatial structure of the nodes.,By comparing the difference ratio (similarity) of the node v connection methods in the node graph structure, that is, the same connection direction / all connection directions,,specifically, statistical behavior i or posture k i Get all the connection directions of node v, then get the connection direction of node v in video frame A, compare them to get the number of the same connection directions, and calculate the similarity xs with the number of all connection directions iv 、xs kiv , that is, the number of connection directions / behaviors i or postures k that are the same as v in video frame A i The number of all connection directions of the next node v; combined with the contribution of node v to different behaviors and posture recognition processes, the similarity score P between the video frame A and the behavior i is obtained i .

[0097] In an optional embodiment of the present application, the behavior type corresponding to the current behavior in the current video frame is obtained according to the similarity score value, and a dynamic special effect corresponding to the behavior type is output at a corresponding position on the user's screen, including:

[0098] The similarity score value of the posture of the current behavior in the current video frame to the corresponding behavior of the first similar video frame data is compared with the second preset threshold, and whether to output dynamic special effects is determined according to the comparison result.

[0099] It should be noted that the similarity score P between the user posture in the current video frame and a certain behavior is obtained by calculating i , according to the behavior similarity score P of the current video frame i, determine the behavior type of the current video frame (the behavior category with the largest similarity score value), and then output the corresponding dynamic special effects at the corresponding position on the user's screen based on the behavior type determined, completing the intelligent recognition of pictures applied to video communication.

[0100] In an optional embodiment of the present application, it includes:

[0101] When the similarity score value is greater than or equal to a second preset threshold, a dynamic special effect corresponding to the behavior type is output at a corresponding position on the user screen;

[0102] When the similarity score value is less than the second preset threshold, no dynamic special effects are output on the user screen.

[0103] It should be noted that if the posture feature vector of the current video frame has a behavior similarity score P with any behavior pattern i When both are less than 0.6, it is considered that it does not belong to any behavior mode and no dynamic effects or virtual props are added.

[0104] See also Figure 3 As shown, an embodiment of the present application discloses an intelligent image recognition device 20 applied to video communication, including a processor 21 and a memory 22; wherein the memory 22 is used to store computer programs; the processor 21 is used to implement an intelligent image recognition method applied to video communication provided by the aforementioned method embodiment by executing the computer program.

[0105] For the specific process of the above-mentioned method for intelligent image recognition applied to video communication, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0106] Furthermore, the memory 22, as a carrier for storing resources, may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0107] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26; wherein the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0108] Furthermore, the present application also discloses a computer-readable storage medium. Figure 4As shown, it is used to store a computer program 31, wherein when the computer program is executed by the processor, an intelligent picture recognition method applied to video communication provided by the aforementioned method embodiment is implemented.

[0109] For the specific process of the above-mentioned method for intelligent image recognition applied to video communication, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0110] An embodiment of the present application also provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer executes an intelligent picture recognition method applied to video communication as shown in an embodiment of the method of the present application.

[0111] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0112] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory, a memory, a read-only memory, an electrically programmable, an electrically erasable programmable, a register, a hard disk, a removable disk, or any other form of storage medium known in the art.

[0113] The above is a detailed introduction to the method, device and medium for intelligent image recognition applied to video communication provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for intelligent image recognition applied to video communication, characterized in that: include: Get video data samples; Extracting features from each video frame in the video data sample to obtain node graph structures and posture vectors of different postures under different behaviors; Performing feature analysis on each video frame data in the video data sample according to the node graph structure and the posture vector to obtain common features and unique features of different postures under different behaviors; According to the common features and unique features of the current behavior and different postures under different behaviors in the current video frame, obtain the first similar video frame data in the video data sample corresponding to the current video frame; Analyze the contribution of each node in the node graph structure of the current behavior in the current video frame to obtain the recognition contribution of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data; According to the recognition contribution degree of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data, a similarity score value of the current behavior in the current video frame to the posture under the behavior corresponding to the first similar video frame data is obtained; According to the similarity score value, the behavior type corresponding to the current behavior in the current video frame is obtained, and the dynamic special effects corresponding to the behavior type are output at the corresponding position on the user's screen.

2. The method for intelligent image recognition applied to video communication according to claim 1, characterized in that: Perform feature extraction on each video frame data in the video data sample to obtain node graph structures and posture vectors of different postures under different behaviors, including: Using a human body detection algorithm to locate the user position in each frame of the video data sample; According to the user position in each frame of the video data sample, a posture estimation algorithm is used to identify key nodes of the human body in each frame, and a node graph structure of different postures under different behaviors is obtained; The key nodes in the node graph structure are vectorized to obtain posture vectors of different postures under different behaviors.

3. The method for intelligent image recognition applied to video communication according to claim 1, characterized in that: Perform feature analysis on each video frame data in the video data sample according to the node graph structure and the posture vector to obtain common features and unique features of different postures under different behaviors, including: The maximum common subgraph of different postures under each same behavior is obtained using the maximum common subgraph acquisition algorithm to obtain the common features of different postures under each same behavior; According to the common features of different postures under each same behavior, the corresponding node graph structure is deleted to obtain the unique features of different postures under each same behavior.

4. The method for intelligent image recognition applied to video communication according to claim 3, characterized in that: Further including: The cosine similarity calculation is performed on the posture vectors corresponding to the common features of different postures under any two different behaviors to obtain the first similarity score of the common features between any two behaviors; The maximum common subgraph between two behaviors whose first similarity scores are greater than a first preset threshold is used to obtain the maximum common subgraph between the two behaviors using a maximum common subgraph acquisition algorithm, and the common features are corrected according to the maximum common subgraph to obtain the common features of different postures under different behaviors.

5. The method for intelligent image recognition applied to video communication according to claim 1, characterized in that: According to the common features and unique features of the current behavior and different postures under different behaviors in the current video frame, the first similar video frame data in the video data sample corresponding to the current video frame is obtained, including: Compare the current behavior in the current video frame with common features of different behaviors to obtain multiple first behaviors corresponding to the current behavior in the current video frame; The current behavior in the current video frame is compared with the unique features of different postures under the multiple first behaviors to obtain the first similar video frame data in the video data sample corresponding to the current video frame.

6. The method for intelligent image recognition applied to video communication according to claim 1, characterized in that: The contribution degree of each node in the node graph structure of the current behavior in the current video frame is analyzed to obtain the recognition contribution degree of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data, wherein the calculation formula of the recognition contribution degree is: Among them, G v represents the recognition contribution of node v in the current behavior j to node v in other behaviors r, f j,v represents the frequency of occurrence of node v in the current behavior j; represents the mean support of all connection modes of node v in the current behavior j; f r,v represents the frequency of occurrence of node v in other behaviors r; represents the mean support of all connection modes of node v in other behaviors r; exp represents an exponential function with the natural constant e as the base.

7. The method for intelligent image recognition applied to video communication according to claim 1, characterized in that: According to the recognition contribution of each node of the current behavior to the posture under the behavior corresponding to the first similar video frame data, a similarity score value of the current behavior in the current video frame to the posture under the behavior corresponding to the first similar video frame data is obtained, wherein the calculation formula of the similarity score value is: Among them, P i Indicates the behavior in the current video frame for the posture k under behavior i i The similarity score value of n and m represents the number of nodes of behavioral features and posture features respectively; (xs iv ×G iv ) represents the matching degree between node v and behavior i; Represents node v and posture k i The degree of matching; xs iv and Represents the node v and behavior i or posture k in the current video frame respectively i Similarity of the next node v; G iv and Respectively represent the behavior i or posture k of node v in the current video frame i The recognition contribution of the next node v; Norm represents the normalization processing function.

8. The method for intelligent image recognition applied to video communication according to claim 1, characterized in that: According to the similarity score value, the behavior type corresponding to the current behavior in the current video frame is obtained, and the dynamic special effects corresponding to the behavior type are output at the corresponding position on the user's screen, including: The similarity score value of the posture of the current behavior in the current video frame to the corresponding behavior of the first similar video frame data is compared with the second preset threshold, and whether to output dynamic special effects is determined according to the comparison result.

9. The method for intelligent picture recognition applied to video communication according to claim 8, characterized in that: Further including: When the similarity score value is greater than or equal to a second preset threshold, a dynamic special effect corresponding to the behavior type is output at a corresponding position on the user screen; When the similarity score value is less than the second preset threshold, no dynamic special effects are output on the user screen.

10. An intelligent image recognition device for video communication, characterized in that: include: A memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, an intelligent picture recognition method applied to video communication as described in any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Single-stage dynamic pose recognition method and device and terminal equipment

    CN113011395A

  • Personnel monitoring recognition method and device, electronic equipment and readable storage medium

    CN114022841A

  • Data processing method and device, electronic equipment and computer storage medium

    CN114241595A

  • User posture recognition method and device, electronic equipment and storage medium

    CN117765564A

  • Personnel behavior abnormity identification method and device, electronic equipment and medium

    CN117877124A