Picture intelligent recognition method and device applied to video communication

Abstract

This invention relates to the field of image communication technology, providing an intelligent image recognition method and apparatus for video communication. The method includes: acquiring video data samples; extracting features from each video frame in the video data samples to obtain node graph structures and pose vectors for different postures under different behaviors; then obtaining common and unique features for different postures under different behaviors; and improving the recognition adaptability for a particular behavior type by combining the common and unique features. Simultaneously, based on existing behavior type features, the contribution degree and similarity of the included body nodes are analyzed to obtain the behavior type corresponding to the current behavior in the current video frame, and the corresponding dynamic effect is output at the corresponding position on the user's screen. This improves the stability, accuracy, and adaptability of recognizing different behaviors.

Description

Technical Field

[0001] This invention relates to the field of image communication technology, and in particular to an intelligent image recognition method and apparatus for video communication. Background Technology

[0002] With the rapid development of 5G networks and video communication technologies, video calls and live streaming have become increasingly integral to people's lives. The application of video communication technology can significantly shorten the distance between users and enrich their daily lives. At the same time, as users' demands for interactivity and entertainment in video communication gradually increase, they are no longer satisfied with simple video calls and live streams. Therefore, by utilizing intelligent image recognition technology to identify users' actions and expressions during video communication and adding corresponding dynamic effects or animations, the goal is to enhance users' sense of participation and interaction during video communication.

[0003] Existing intelligent image recognition methods typically perform facial recognition, expression tracking, and gesture recognition on people in videos. Based on the user's posture, they dynamically add virtual decorations and effects to enhance user engagement and interactivity. However, in practical applications, some target behaviors can be achieved through various different postures. Due to differences in users' movement habits and physical conditions, their postures when acquiring dynamic effects will vary during video calls or live streams. This discrepancy between the user's posture and the target behavior stored in the system can lead to posture recognition failures, resulting in poor system stability, accuracy, and adaptability. Summary of the Invention

[0004] In view of the above problems, this application provides an intelligent image recognition method and device for video communication, which can improve the stability, accuracy and adaptability of recognizing different behaviors.

[0005] In a first aspect, embodiments of this application provide an intelligent image recognition method for video communication, including:

[0006] Obtain video data samples;

[0007] Feature extraction is performed on each video frame in the video data sample to obtain the node graph structure and pose vector for different behaviors;

[0008] Based on the node graph structure and pose vector, feature analysis is performed on each video frame in the video data sample to obtain the common features and unique features of different poses under different behaviors.

[0009] Based on the common and unique features of the current behavior and different postures under different behaviors in the current video frame, the first similar video frame data in the video data sample corresponding to the current video frame is obtained;

[0010] The contribution of each node in the node graph structure of the current behavior in the current video frame is analyzed to obtain the contribution of each node of the current behavior to the recognition of pose under the corresponding behavior in the first similar video frame data.

[0011] Based on the contribution of each node of the current behavior to the recognition of the pose under the corresponding behavior in the first similar video frame data, the similarity score of the current behavior in the current video frame to the pose under the corresponding behavior in the first similar video frame data is obtained.

[0012] Based on the similarity score, the behavior type corresponding to the current behavior in the current video frame is obtained, and the corresponding dynamic effect is output at the corresponding position on the user's screen.

[0013] In one possible implementation, feature extraction is performed on each video frame in the video data sample to obtain the node graph structure and pose vector for different poses under different behaviors, including:

[0014] Human detection algorithms are used to locate the user's position in each frame of video data samples.

[0015] Based on the user position in each frame of the video data sample, the pose estimation algorithm is used to identify the key nodes of the human body in each frame, and the node graph structure of different poses under different behaviors is obtained.

[0016] Vectorize the key nodes in the node graph structure to obtain pose vectors for different behaviors.

[0017] In one possible implementation, feature analysis is performed on each video frame in the video data sample based on the node graph structure and pose vector to obtain common features and unique features of different poses under different behaviors, including:

[0018] The maximum common subgraph for each different pose under the same behavior is obtained using the maximum common subgraph acquisition algorithm, thus obtaining the common features of different poses under the same behavior;

[0019] Based on the common features of different poses under the same behavior, the corresponding node graph structure is reduced to obtain the unique features of different poses under the same behavior.

[0020] One possible implementation includes:

[0021] The cosine similarity is calculated for the pose vectors corresponding to the common features of different poses under any two different behaviors, and the first similarity score of the common features between any two behaviors is obtained.

[0022] For two behaviors with a first similarity score greater than a first preset threshold, the maximum common subgraph between the two behaviors is obtained using the maximum common subgraph acquisition algorithm. The common features are then corrected based on the maximum common subgraph to obtain the common features of different postures under different behaviors.

[0023] In one possible implementation, based on the common and unique features of the current behavior and different poses under different behaviors in the current video frame, the first similar video frame data in the video data sample corresponding to the current video frame is obtained, including:

[0024] By comparing the current behavior in the current video frame with the common features of different behaviors, multiple first behaviors corresponding to the current behavior in the current video frame are obtained;

[0025] The current action in the current video frame is compared with the unique features of different poses under multiple first actions to obtain the first similar video frame data in the video data sample corresponding to the current video frame.

[0026] In one possible implementation, the contribution of each node in the node graph structure of the current behavior in the current video frame is analyzed to obtain the recognition contribution of each node of the current behavior to the pose recognition under the corresponding behavior in the first similar video frame data. The formula for calculating the recognition contribution is as follows:

[0027]

[0028] Among them, G v f represents the degree of contribution of node v in the current behavior j to the identification of node v in other behaviors r. j,v This indicates the frequency of node v in the current behavior j; f represents the mean support of all connection methods for node v in the current action j; r,v This indicates the frequency of occurrence of node v in other behaviors r; represents the mean support of all connection methods of node v in other behaviors r; exp represents an exponential function with base e.

[0029] In one possible implementation, the similarity score of the current behavior in the current video frame to the pose of the corresponding behavior in the first similar video frame is obtained based on the contribution of each node of the current behavior to the pose recognition of the corresponding behavior in the first similar video frame data. The formula for calculating the similarity score is as follows:

[0030]

[0031] Among them, P i This indicates that the pose k in the current video frame represents the action i. iThe similarity score, where n and m represent the number of nodes for behavioral features and pose features, respectively; (xs iv ×G iv (xs) represents the degree of matching between node v and behavior i; kiv +G kiv ) represents the relationship between node v and pose k i The degree of matching; xs iv and XS kiv Representing node v and behavior i or pose k in the current video frame, respectively. i Similarity of the next node v; G iv and G kiv These represent the relationships between node v in the current video frame and action i or pose k, respectively. i The recognition contribution of the next node v; Norm represents the normalization function.

[0032] In one possible implementation, the behavior type corresponding to the current behavior in the current video frame is obtained based on the similarity score, and the dynamic effect corresponding to the behavior type is output at the corresponding position on the user's screen, including:

[0033] The similarity score of the current behavior in the current video frame to the pose of the corresponding behavior in the first similar video frame data is compared with the second preset threshold, and the dynamic effect is output based on the comparison result.

[0034] One possible implementation includes:

[0035] When the similarity score is greater than or equal to the second preset threshold, the dynamic effect corresponding to the behavior type is output at the corresponding position on the user's screen;

[0036] When the similarity score is less than the second preset threshold, no dynamic effects are output on the user's screen.

[0037] Secondly, embodiments of this application provide an intelligent image recognition device for video communication, including a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it implements the various possible implementations of the first aspect.

[0038] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the various possible implementations of the first aspect.

[0039] The beneficial effects of this application are as follows: This application can obtain the common features and unique features of postures under each behavior type through in-depth analysis. By combining the common features and unique features, the recognition adaptability for a behavior type is improved. At the same time, based on the existing behavior type features, the contribution degree and similarity of the body nodes included are analyzed. That is, different behaviors have different focuses for node analysis, which improves the stability, accuracy and adaptability of the recognition of different behaviors. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the steps of an intelligent image recognition method for video communication provided in an embodiment of this application.

[0041] Figure 2 This is a schematic diagram of the maximum common subgraph in a two-node graph structure provided in an embodiment of this application;

[0042] Figure 3 This is a structural block diagram of an intelligent image recognition device for video communication provided in an embodiment of this application;

[0043] Figure 4 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0044] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0045] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0046] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0047] During video calls or live streams, users typically make their usual gestures when they want to obtain the desired dynamic effects. Since some target behaviors can be achieved through various different gestures, and different users have different gesture habits, their gestures when obtaining dynamic effects will also differ during video calls or live streams. This can lead to situations where the user's gesture does not match the target behavior stored in the system, resulting in recognition failures and poor stability, accuracy, and adaptability of the image intelligent recognition system.

[0048] This application analyzes different postures of a certain behavior to obtain the common node spatial distribution features and unique node spatial distribution features among different postures, assigns different behavior recognition contribution values ​​to different nodes, and finally improves the ability to recognize user postures in images during video communication by calculating the similarity values ​​of different behaviors.

[0049] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0050] See Figure 1 As shown in the figure, this application discloses an intelligent image recognition method for video communication, including:

[0051] Step S11: Obtain video data samples;

[0052] Step S12: Extract features from each video frame in the video data sample to obtain the node graph structure and pose vector for different poses under different behaviors.

[0053] Step S13: Perform feature analysis on each video frame data in the video data sample based on the node graph structure and pose vector to obtain the common features and unique features of different poses under different behaviors.

[0054] Step S14: Based on the common and unique features of the current behavior and different poses under different behaviors in the current video frame, obtain the first similar video frame data in the video data sample corresponding to the current video frame;

[0055] Step S15: Analyze the contribution of each node in the node graph structure of the current behavior in the current video frame to obtain the contribution of each node of the current behavior to the recognition of the pose under the corresponding behavior in the first similar video frame data.

[0056] Step S16: Based on the degree of contribution of each node of the current behavior to the recognition of the pose under the corresponding behavior of the first similar video frame data, obtain the similarity score of the current behavior in the current video frame to the pose under the corresponding behavior of the first similar video frame data.

[0057] Step S17: Based on the similarity score, obtain the behavior type corresponding to the current behavior in the current video frame, and output the dynamic effect corresponding to the behavior type at the corresponding position on the user's screen.

[0058] This process involves collecting necessary video data through multiple channels, specifically videos of actions triggered by preset dynamic effects or animations. The collected videos include those of different ages, skin tones, and environmental poses, aiming to ensure the comprehensiveness of the collected video data samples. Specifically, the types of actions to be collected are determined, such as the peace sign, heart gesture, and thumbs-up. Based on these types, relevant video data samples are collected from multiple sources. The collected video data samples undergo preprocessing, including noise reduction and enhancement. Finally, video data samples of different poses corresponding to the same action are categorized and labeled as X. 11 (i.e., behavior 1, pose 1), and other video data samples follow the same pattern, such as X. 21 X 22 (i.e., behavior 2 pose 1, behavior 2 pose 2), the processed video data samples belonging to the same behavior are stored in a database, and a set of video data samples of all different behaviors is constructed to complete the collection of the required data samples.

[0059] In the above embodiment steps, video data samples are first acquired, and features are extracted from each video frame in the video data samples to obtain the node graph structure and pose vectors for different postures under different behaviors. Then, the common features and unique features of different postures under different behaviors are obtained. Next, the recognition contribution degree of each node of the current behavior to the posture of the corresponding behavior in the first similar video frame data and the similarity score of the current behavior in the current video frame to the posture of the corresponding behavior in the first similar video frame data are obtained. Finally, based on the common features, unique features, recognition contribution degree, and similarity score, the behavior type corresponding to the current behavior in the current video frame is obtained, and the dynamic effect corresponding to the behavior type is output at the corresponding position on the user's screen. This application can obtain the common features and unique features of the posture under each behavior type by deeply analyzing each behavior type. By combining the common features and unique features, the recognition adaptability for a behavior type is improved. At the same time, based on the existing behavior type features, the contribution degree and similarity of the body nodes included are analyzed, that is, different behaviors have different focuses for node analysis, which improves the recognition stability, accuracy, and adaptability for different behaviors.

[0060] It should be noted that in the following description, "behavior" refers to the triggering condition of a certain special effect. For example, making a heart shape with your hands is an behavior, while "posture" refers to the way of performing this behavior, such as making a heart shape with your fingers, making a heart shape with both hands, making a heart shape with both arms, etc.

[0061] Since some behaviors can be achieved through multiple different postures, this indicates that different postures for the same behavior share certain similarities, primarily in the similarity of the human body's node spatial structure. However, different postures also exhibit differences in their node spatial structure, allowing for differentiation. To obtain the common and unique features among different postures for the same behavior, it is necessary to acquire the node graph structure of each sample. By comparing these structures, the similarities and differences between different postures can be determined, ultimately extracting the different posture features for different behaviors.

[0062] In an optional embodiment of this application, feature extraction is performed on each video frame in the video data sample to obtain the node graph structure and pose vector for different poses under different behaviors, including:

[0063] Human detection algorithms are used to locate the user's position in each frame of video data samples.

[0064] Based on the user position in each frame of the video data sample, the pose estimation algorithm is used to identify the key nodes of the human body in each frame, and the node graph structure of different poses under different behaviors is obtained.

[0065] Vectorize the key nodes in the node graph structure to obtain pose vectors for different behaviors.

[0066] It should be noted that the node graph structure and pose feature vector can reflect the specific behavioral characteristics of the user in the image. First, the human pose features in each video data sample are extracted using existing algorithms to obtain the feature vectors and node graph structures of different poses under different behaviors.

[0067] Specifically, firstly, human detection algorithms (such as Haar Cascades Classifier, HOG+SVM algorithm, etc.) are used to locate the user position in each frame of the video data sample; based on the known user body position in the video frame, pose estimation algorithms (such as OpenPose, AlphaPose algorithm, etc.) are used to identify the key nodes of the human body in each frame, and the key nodes are connected in combination with the human body structure to obtain a node graph structure; the user key point features (node ​​graph structure) obtained above are vectorized to obtain the pose vector of each video data sample in the current behavior type; repeat the above steps to obtain the pose vector of video data samples with all poses under different behaviors, and complete the feature extraction.

[0068] It should be noted that each pose vector has the same length (containing all human nodes) and the corresponding dimensions have the same meaning. Each element corresponds to the position or angle information of a key node in the image.

[0069] In an optional embodiment of this application, feature analysis is performed on each video frame in the video data sample based on the node graph structure and pose vector to obtain common features and unique features of different poses under different behaviors, including:

[0070] The maximum common subgraph for each different pose under the same behavior is obtained using the maximum common subgraph acquisition algorithm, thus obtaining the common features of different poses under the same behavior;

[0071] Based on the common features of different poses under the same behavior, the corresponding node graph structure is reduced to obtain the unique features of different poses under the same behavior.

[0072] It should be noted that the maximum common subgraph refers to the largest subgraph that can be found in two or more graph structures. This subgraph exists in all the graphs being compared. For two input graphs with different nodes (called graph pairs), the maximum common subgraph is detected (as shown in the attached figure). Figure 2 As shown), and this subgraph is required to be connected and isomorphic, such as... Figure 2 As shown. Therefore, the node graph structure of different poses under the same behavior must have a maximum common subgraph. This maximum common subgraph is the common feature of the current behavior, while the node information outside the maximum common subgraph belongs to the unique feature of each pose.

[0073] During video communication, the possible behavior types of the image are identified by searching the maximum common subgraph of a certain behavior in the video frame image. Then, the search is carried out according to the unique features of different poses under each behavior type, so as to realize the intelligent recognition of video frame images.

[0074] Specifically, firstly, the existing Maximum Common Subgraph Acquisition Algorithm (MCSA) is used to acquire the maximum common subgraph for behavior i, denoted as GT. i The node graph structure represents the common features of behavior i; subsequently, the unique features of any pose under behavior i are extracted, and the maximum common subgraph GT is used. i Each node graph structure under pose k is reduced to obtain the unique node graph structure of all samples under pose k; the above steps are repeated to obtain the common features and unique features of different poses under different behaviors, and all common features and unique features are vectorized.

[0075] It should be noted that, in order to ensure the accuracy of user behavior recognition, that is, the uniqueness of the common features of each behavior, it is also necessary to analyze the similarity of common features between different behaviors. When the similarity of common features between different behaviors is high, it is necessary to analyze and reduce the common features of the behavior type by the maximum common subgraph, so as to ensure the stability of image recognition.

[0076] In an optional embodiment of this application, it includes:

[0077] The cosine similarity is calculated for the pose vectors corresponding to the common features of different poses under any two different behaviors, and the first similarity score of the common features between any two behaviors is obtained.

[0078] For two behaviors with a first similarity score greater than a first preset threshold, the maximum common subgraph between the two behaviors is obtained using the maximum common subgraph acquisition algorithm. The common features are then corrected based on the maximum common subgraph to obtain the common features of different postures under different behaviors.

[0079] Specifically, in this embodiment, the similarity score μ of the common feature vector of behavior i with all other behaviors is first calculated. u (Cosine similarity), which is the similarity score of the vector with the u-th behavior; a threshold is set to filter the similarity scores. When the vector similarity ε between any two behaviors is greater than 0.5 (first preset threshold), the common features of the two behaviors are considered too similar; the maximum common subgraph of the two behaviors is obtained again, and the subgraphs of the two behaviors are corrected using the obtained maximum common subgraph to obtain the corrected node graph structure GT′ of behavior i. i That is, the common features of the final behavior i.

[0080] It should be further explained that the correction method here is to compare and filter the newly acquired common subgraph with the original behavior feature subgraph to obtain the behavior feature subgraph with the common subgraph removed, which is the common feature of this behavior.

[0081] In an optional embodiment of this application, based on the common and unique features of the current behavior and different poses under different behaviors in the current video frame, the first similar video frame data in the video data sample corresponding to the current video frame is obtained, including:

[0082] By comparing the current behavior in the current video frame with the common features of different behaviors, multiple first behaviors corresponding to the current behavior in the current video frame are obtained;

[0083] The current action in the current video frame is compared with the unique features of different poses under multiple first actions to obtain the first similar video frame data in the video data sample corresponding to the current video frame.

[0084] It should be noted that in the process of intelligent image recognition, the common features (node ​​graph structure) of video frames and different behaviors are first compared to determine the possible behavior type of the video frame (such as peace sign, heart gesture, like, etc.). Then, the specific features of different postures under the behavior type are compared to determine the posture of the video frame.

[0085] However, when comparing video frames with feature node graph structures, different behaviors and different poses under different behavior types will inevitably contain the same nodes in their node graph structures, and there may still be local similarities. Therefore, in order to improve the adaptability of intelligent image recognition, we analyze the recognition contribution of different nodes in the node graph structure, so that special nodes have higher weights in the image recognition process.

[0086] In an optional embodiment of this application, a contribution analysis is performed on each node in the node graph structure of the current behavior in the current video frame to obtain the recognition contribution of each node of the current behavior to the pose recognition of the corresponding behavior in the first similar video frame data. The formula for calculating the recognition contribution is as follows:

[0087]

[0088] Among them, G v f represents the degree of contribution of node v in the current behavior j to the identification of node v in other behaviors r. j,v This indicates the frequency of node v in the current behavior j; f represents the mean support of all connection methods for node v in the current action j; r,v This indicates the frequency of occurrence of node v in other behaviors r; represents the mean support of all connection methods of node v in other behaviors r; exp represents an exponential function with base e.

[0089] Specifically, in the embodiments of this application, the node graph structures of different behaviors are first compared and analyzed to obtain the frequency f of node v in a certain behavior or pose feature. v (Number of node graph structures / Number of node graph structures containing node v). The lower the frequency of node v, the higher its contribution. The lower the frequency of node v under other behaviors, the higher the frequency of node v under the current behavior, and thus the higher its contribution. This indicates that the more unique node v is under the current behavior, the higher its contribution. Subsequently, the connection relationships between nodes are analyzed. Frequent itemsets connected to node v are obtained using data mining algorithms (such as FP-Growth algorithm). The overall support ZC of the frequent itemsets r corresponding to node v is calculated. v,rThe lower the support, the higher the contribution of node v. The definition of frequent itemsets is unique to the FP-Growth algorithm, and the calculation of support is similar, both belonging to existing algorithms and definitions. It represents the frequency of occurrence of different connection methods in the node connection process. The lower the support of a node's frequent itemsets in all behaviors, and the higher the support in the current behavior, the higher its contribution.

[0090] It should be further noted that when analyzing the support of a node to determine its contribution, it is necessary to analyze all frequent itemsets corresponding to a node and calculate their mean as the parameter affecting the contribution (mean support).

[0091] Using the above formula for calculating the degree of recognition contribution, we can traverse each human node according to the node graph structure of different behaviors and postures, and obtain the degree of recognition contribution of each node to different behaviors and postures.

[0092] In an optional embodiment of this application, based on the contribution of each node of the current behavior to the recognition of the pose under the corresponding behavior in the first similar video frame data, the similarity score of the current behavior in the current video frame to the pose under the corresponding behavior in the first similar video frame data is obtained, wherein the formula for calculating the similarity score is:

[0093]

[0094] Among them, P i This indicates that the pose k in the current video frame represents the action i. i The similarity score, where n and m represent the number of nodes for behavioral features and pose features, respectively; (xs iv ×G iv The ) represents the degree of matching between node v and behavior i; Representing node v and pose k i The degree of matching; xs iv and Representing node v and behavior i or pose k in the current video frame, respectively. i Similarity of the next node v; G iv and These represent the relationships between node v in the current video frame and action i or pose k, respectively. i The recognition contribution of the next node v; Norm represents the normalization function.

[0095] It should be noted that the process involves comparing and analyzing the user's behavior and posture in the video frame, calculating the similarity score between the node graph structure and the feature node graph structure in the video frame, and correcting the similarity score by taking into account the contribution of nodes to different behaviors and postures during the comparison process (the correction process involves calculating the similarity between the user and different postures under the current behavior when calculating the behavior similarity), thus obtaining the similarity score value between the user and different behaviors in the video frame.

[0096] Specifically, in this embodiment, firstly, the sampling interval of the video frames in the video communication is 2 seconds (which can be set according to specific circumstances, such as keyframe analysis), the current video frame A is obtained, and the node graph structure corresponding to video frame A is obtained; then, the node graph structure of video frame A is compared with that of behavior and pose features, and the similarity xs between the connection methods of the nodes v of the behavior and pose features is obtained. iv xs kiv Similarity is the comparison of differences in the structure (features) of a node graph, such as different node connection methods, different contained nodes, and different spatial structures formed by the nodes. It is achieved by comparing the ratio of differences in the connection methods of nodes v in the node graph structure (similarity), i.e., same connection direction / all connection directions. Specifically, it is statistically analyzed using behavior i or pose k. i Take all connection directions of node v, then obtain the connection directions of node v in video frame A, compare them to get the number of identical connection directions, and calculate the similarity xs by comparing the number of all connection directions. iv xs kiv That is, the number of connections with the same direction as v in video frame A / behavior i or pose k i The number of all connection directions of the next node v; combined with the contribution of node v to the recognition of different behaviors and poses, obtain the similarity score P between video frame A and behavior i. i .

[0097] In an optional embodiment of this application, the behavior type corresponding to the current behavior in the current video frame is obtained based on the similarity score, and the dynamic effect corresponding to the behavior type is output at the corresponding position on the user's screen, including:

[0098] The similarity score of the current behavior in the current video frame to the pose of the corresponding behavior in the first similar video frame data is compared with the second preset threshold, and the dynamic effect is output based on the comparison result.

[0099] It should be noted that the similarity score P between the user's pose and a certain behavior in the current video frame is calculated. i Based on the behavioral similarity score P of the current video frame iIt determines the behavior type of the current video frame (the behavior category with the highest similarity score), and then outputs the corresponding dynamic effects at the corresponding position on the user's screen based on the determined behavior type, thus completing the intelligent image recognition applied to video communication.

[0100] In an optional embodiment of this application, it includes:

[0101] When the similarity score is greater than or equal to the second preset threshold, the dynamic effect corresponding to the behavior type is output at the corresponding position on the user's screen;

[0102] When the similarity score is less than the second preset threshold, no dynamic effects are output on the user's screen.

[0103] It should be noted that if the pose feature vector of the current video frame has a similarity score P to any behavior pattern, then... i If all values ​​are less than 0.6, it is considered not to belong to any behavior pattern, and no dynamic effects or virtual props will be added.

[0104] See Figure 3 As shown in the figure, this application discloses an intelligent image recognition device 20 for video communication, including a processor 21 and a memory 22; wherein, the memory 22 is used to store computer programs; the processor 21 is used to implement the intelligent image recognition method for video communication provided in the aforementioned method embodiment by executing the computer programs.

[0105] For details regarding the specific process of the above-mentioned intelligent image recognition method applied to video communication, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0106] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0107] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The power supply 23 is used to provide operating voltage for the various hardware devices on the electronic device 20. The communication interface 24 can create a data transmission channel between the electronic device 20 and external devices. The communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here. The input / output interface 25 is used to acquire external input data or output data to the outside world. Its specific interface type can be selected according to the specific application needs, and is not specifically limited here.

[0108] Furthermore, embodiments of this application also disclose a computer-readable storage medium, such as... Figure 4As shown, a computer program 31 is used to store the computer program, which, when executed by a processor, implements the intelligent image recognition method for video communication provided in the aforementioned method embodiment.

[0109] For details regarding the specific process of the above-mentioned intelligent image recognition method applied to video communication, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0110] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute an intelligent image recognition method for video communication as shown in the method embodiment of this application.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0112] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory, memory, read-only memory, electrically programmable, electrically erasable programmable, registers, hard disks, removable disks, or any other form of storage medium known in the art.

[0113] The above provides a detailed description of an intelligent image recognition method, device, and medium for video communication provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for intelligently identifying pictures applied to video communication, characterized in that, The method comprises the following steps: acquiring video data samples; extracting features of each video frame data in the video data samples to obtain node graph structures and posture vectors of different postures under different behaviors, the behaviors being used to represent trigger conditions of special effects, the postures being used to represent manners of performing behaviors, each posture vector having the same length and the same meaning in corresponding dimensions, and each element in each posture vector corresponding to position or angle information of a key node in a picture; performing feature analysis on each video frame data in the video data samples according to the node graph structures and the posture vectors to obtain common features and unique features of different postures under different behaviors; obtaining first similar video frame data in the video data samples corresponding to the current video frame according to a current behavior in the current video frame and the common features and the unique features of different postures under different behaviors; performing contribution degree analysis on each node in the node graph structure of the current behavior in the current video frame to obtain recognition contribution degrees of each node of the current behavior to postures under behaviors corresponding to the first similar video frame data, the calculation formula of the recognition contribution degrees being: wherein, representing the current behavior the node for other behaviors the node the degree of identification contribution, representing the current behavior the node the frequency of occurrence; representing the current behavior the node the average support of all connection modes; representing other behaviors the node the frequency of occurrence; representing other behaviors the node the average support of all connection modes; representing the exponential function with the natural constant e as the base; obtaining a similarity score value of the current behavior in the current video frame to postures under behaviors corresponding to the first similar video frame data according to the recognition contribution degrees of each node of the current behavior to the postures under the behaviors corresponding to the first similar video frame data; when the similarity score value is greater than or equal to a second preset threshold, obtaining a behavior type corresponding to the current behavior in the current video frame and outputting a dynamic special effect corresponding to the behavior type at a corresponding position on a user screen, and when the similarity score value is less than the second preset threshold, not outputting any dynamic special effect on the user screen; the similarity score value satisfying the following formula: in, Indicates the behavior in the current video frame relative to the behavior. Lower position The similarity score, and These represent the number of nodes representing behavioral features and pose features, respectively. Represents a node With behavior The degree of matching; Represents a node With posture The degree of matching; and These represent the nodes in the current video frame. With behavior or posture Next node Similarity; and These represent the nodes in the current video frame. Regarding behavior or posture Next node The contribution of identification; This represents the normalization function.

2. The picture intelligent recognition method applied to video communication according to claim 1, characterized in that, extracting features of each video frame data in the video data samples to obtain node graph structures and posture vectors of different postures under different behaviors, comprising: positioning a user position of each frame in the video data samples by using a human body detection algorithm; identifying key nodes of a human body in each frame by using a pose estimation algorithm according to the user position of each frame in the video data samples to obtain node graph structures of different postures under different behaviors; vectorizing the key nodes in the node graph structures to obtain posture vectors of different postures under different behaviors.

3. The picture intelligent recognition method applied to video communication according to claim 1, characterized in that, performing feature analysis on each video frame data in the video data samples according to the node graph structures and the posture vectors to obtain common features and unique features of different postures under different behaviors, comprising: obtaining a maximum common subgraph of each different posture under the same behavior by using a maximum common subgraph obtaining algorithm to obtain common features of each different posture under the same behavior; pruning the corresponding node graph structure according to the common features of each different posture under the same behavior to obtain unique features of each different posture under the same behavior.

4. The picture intelligent recognition method applied to video communication according to claim 3, characterized in that, further comprising: performing cosine similarity calculation on posture vectors corresponding to common features of different postures under any two different behaviors to obtain a first similarity score of the common features between any two behaviors; The maximum common subgraph between the two behaviors is obtained by using a maximum common subgraph obtaining algorithm for the maximum common subgraph of the two behaviors with the first similarity score greater than the first preset threshold, and the common features are corrected according to the maximum common subgraph to obtain the common features of different postures under different behaviors.

5. The picture intelligent recognition method applied to video communication according to claim 1, characterized in that, According to the common features of different postures under different behaviors and the unique features of the current behavior in the current video frame, first similar video frame data in a video data sample corresponding to the current video frame is obtained, including: The common features of the current behavior and different behaviors in the current video frame are compared to obtain a plurality of first behaviors corresponding to the current behavior in the current video frame; The unique features of different postures under the current behavior and the plurality of first behaviors in the current video frame are compared to obtain the first similar video frame data in the video data sample corresponding to the current video frame.

6. A picture intelligent recognition device applied to video communication, characterized in that, It includes: A memory and a processor, the memory stores executable code, and the processor executes the executable code to implement the picture intelligent recognition method applied to video communication according to any one of claims 1-5.

Images