Behavior recognition method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a behavior recognition method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first video; the first video comprises at least two interaction objects; based on a first target model and the first video, generating predicted interaction behavior information for the at least two interaction objects; target information for the interaction object is generated based on the predicted interaction behavior information and the happened behavior information of the at least one interaction object, and the target information can be displayed on a display screen or played in a voice mode; the occurred behavior information is obtained based on the first video.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of image recognition technology, and in particular to a behavior recognition method, device, equipment, and storage medium. Background Art

[0002] Traditional methods for identifying human behavior that completely rely on image features only input video streams, extract image-level features from human behavior, and then determine specific categories. This method can only determine actions that have already occurred, and cannot warn of behaviors that have not yet occurred. In addition, it only considers single video form input, which has limited effect when processing human interactions with different objects. Summary of the invention

[0003] In view of this, the embodiments of the present application at least provide a behavior recognition method, apparatus, device, and storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a behavior recognition method, comprising obtaining a first video; the first video comprises at least two interactive objects; based on a first target model and the first video, generating predicted interactive behavior information between at least two interactive objects; based on the predicted interactive behavior information and information on actual behaviors of at least one interactive object, generating target information for the interactive objects, wherein the target information can be displayed on a display screen or played by voice; the information on actual behaviors is obtained based on the first video.

[0006] In a second aspect, an embodiment of the present application provides a behavior recognition device, which includes: an acquisition module for obtaining a first video; the first video includes at least two interactive objects; a first generation module for generating predicted interactive behavior information between at least two interactive objects based on a first target model and the first video; a second generation module for generating target information for the interactive objects based on the predicted interactive behavior information and actual behavior information of at least one interactive object, wherein the target information can be displayed on a display screen or played by voice; the actual behavior information is obtained based on the first video.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above method.

[0010] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 A schematic diagram of an implementation flow of a behavior recognition method provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of an implementation flow of a behavior recognition method provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of an implementation flow of a behavior recognition method provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of an implementation flow of a behavior recognition method provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of an implementation flow of a behavior recognition method provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of the implementation process of a method for identifying human behavior provided in an embodiment of the present application;

[0018] Figure 7 A schematic diagram of the implementation process of a method for identifying human behavior provided in an embodiment of the present application;

[0019] Figure 8 A schematic diagram of a video frame provided in an embodiment of the present application;

[0020] Fig. 9 A schematic diagram of an implementation flow of a behavior recognition method provided in an embodiment of the present application;

[0021] Fig.10 A schematic diagram of an implementation flow of a method for determining an interactive object and an interactive person provided in an embodiment of the present application;

[0022] Fig.11 A schematic diagram of an implementation flow of an interactive behavior prediction method provided in an embodiment of the present application;

[0023] Fig.12 A schematic diagram of the prediction results provided in the embodiments of the present application;

[0024] Fig.13 A schematic diagram of the structure of a behavior recognition device provided in an embodiment of the present application;

[0025] Fig.14 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0027] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0029] For traditional methods of human behavior recognition that completely rely on image features, only video streams are input, image-level features of human behavior are extracted, and then specific categories are determined. This method can only judge actions that have occurred, and cannot warn of actions that have not occurred. And only considering a single video form input, the effect is limited when dealing with human interaction with different objects. Among them, the method that simply uses video sequence input can only judge actions that have occurred, and cannot give a warning of dangerous behaviors that have not occurred; the method using a large language model requires multiple guidance through prompt words before a specific area description can be given. With the development of large language models, their logical reasoning ability has been certified in different scenarios and tasks, and can be applied to the task of describing and understanding image content. However, this type of model needs to be guided by instruction words for multiple interactions before it can output the corresponding content, and there is no way to directly output recognition results for specific target areas.

[0030] Therefore, the embodiment of the present application proposes a multimodal input method that combines an image processing model with a large language model. By comprehensively utilizing the image processing model's ability to obtain image-level features and the large language model's ability to make logical judgments, taking the same part of the output results of the two, and identifying and warning dangerous behaviors that have not occurred, the final effect can be significantly improved. In the processing process, processes such as removing all backgrounds and removing objects that cannot interact with people are added to reduce interference with the results. The common part of the image model judgment result and the large language model judgment result is used as the output, and information of different dimensions is fully considered to improve the robustness of the results. Compared with the traditional method of simply using video sequence input to understand the behavioral intentions of people, a new structure that combines image features and the logical reasoning results of a large language model is designed, and multimodal input is used to judge the behavior of people.

[0031] The present application embodiment provides a behavior recognition method, which can be executed by a processor of an electronic device. The electronic device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities.

[0032] Figure 1 A schematic diagram of the implementation flow of a behavior recognition method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S103, combining Figure 1 The steps shown are explained.

[0033] Step S101, obtaining a first video; the first video includes at least two interactive objects.

[0034] In some embodiments, the first video can be shot by a shooting device and then saved; it can also be a real-time video; illustratively, the first video is a surveillance video for a certain area, which is not specifically limited in this application.

[0035] In some embodiments, the at least two interactive objects of the first video may be at least two interactive objects in each video frame of the first video; illustratively, the interactive objects may be persons; or objects.

[0036] Step S102: Generate predicted interaction behavior information between the at least two interaction objects based on the first target model and the first video.

[0037] In some embodiments, the first target model is a trained interaction behavior prediction model; it may include a trained person-object interaction behavior prediction model, a trained person-person interaction behavior prediction model, and a trained object-object interaction behavior prediction model; wherein the first target model is used to screen the categories of at least two interaction objects in the first video to obtain at least two interaction objects that can interact; and to predict at least two interaction objects that can interact to obtain predicted interaction behavior information.

[0038] In some embodiments, a trained interactive behavior prediction model can be trained in the following manner: obtaining multiple training samples; each training sample includes video data carrying annotation information; wherein the video data includes at least two interactive objects; there is at least one interactive behavior between at least two interactive objects; interactive behavior can occur between at least two interactive objects; the annotation information includes the category, speed, direction, size of each interactive object and the distance, angle, etc. between at least two interactive objects; and the interaction results corresponding to the video data carrying the annotation information; the interaction results include the type of interactive behavior of at least two interactive objects in the video data and the video clip corresponding to the interactive behavior; the video data carrying the annotation information in each training sample is input into the interactive behavior prediction model to be trained to obtain a prediction result; the prediction result includes the predicted interactive behavior and the predicted video clip corresponding to the interactive behavior; the model parameters of the interactive behavior prediction model to be trained are adjusted based on the loss value between the prediction result and the interactive result until the convergence condition is met, and the trained interactive behavior prediction model is output.

[0039] In some embodiments, the first target model may include a time series model (e.g., an autoregressive integrated moving average model, a generalized autoregressive conditional heteroskedasticity model, etc.), a machine learning model (e.g., a vector machine model, a random forest model, etc.), a deep learning model (a convolutional neural network model, a recurrent neural network model, a self-attention mechanism neural network model, etc.).

[0040] In some embodiments, the predicted interaction behavior information may be interaction behaviors occurring between two or more interaction objects; wherein the predicted interaction behavior information may be interaction behaviors occurring between people, interaction behaviors occurring between people and objects, and interaction behaviors occurring between objects.

[0041] Exemplarily, the predicted interactive behavior information may be an interactive behavior of a handshake between two persons, an interactive behavior of a person picking up a pair of scissors, an interactive behavior of a basketball flying into a basket, and the like.

[0042] In some embodiments, based on the first target model and the first video, the predicted interactive behavior information between the at least two interactive objects is generated; it is understandable that the at least two interactive objects in the first video are input into the trained interactive behavior prediction model to obtain at least one interactive behavior related to the at least two interactive objects and the probability of each interactive behavior occurring between the at least two interactive objects, and the interactive behavior with the highest probability of occurring between the at least two interactive objects is determined as the predicted interactive behavior information. Among them, inputting the at least two interactive objects in the first video into the trained interactive behavior prediction model can also obtain the score of each interactive behavior, and the interactive behavior with the highest score in the at least one interactive behavior related to the at least two interactive objects is determined as the predicted interactive behavior information.

[0043] Exemplarily, at least two interactive objects are taken as a person, a stone, and a football to illustrate the embodiment of the present application. Among them, at least one interactive behavior related to at least two interactive objects includes a person kicking a football, a person kicking a stone, a football hitting a stone, a person picking up a football, a person picking up a stone, etc., wherein the probability of a person kicking a football is 40%, the probability of a person kicking a stone is 10%, the probability of a football hitting a stone is 15%, the probability of a person picking up a football is 15%, and the probability of a person picking up a stone is 20%, so that the person kicking a football with the highest probability is determined as the predicted interactive behavior information.

[0044] Step S103: Generate target information for the interactive object based on the predicted interactive behavior information and the actual behavior information of at least one of the interactive objects.

[0045] Among them, the target information can be displayed on a display screen or played by voice; the information about the behavior that has occurred is obtained based on the first video.

[0046] In some embodiments, the information on the actions that have occurred may be the actions that have occurred on each interactive object itself; for example, if the interactive object is a person, the information on the actions that have occurred may be the person raising his hand, etc.; the information on the actions that have occurred may also be the interactive actions that have occurred between multiple interactive objects; for example, the interactive objects include a person and a door handle, and the actions that have occurred may be the person touching the door handle, etc.

[0047] In some embodiments, the target information is used to display the predicted interaction behavior information and the behavior information of each interactive object on a display screen or play it by voice when the predicted interaction behavior information of the interactive object belongs to the target type; wherein the target type may include dangerous types of interaction behaviors. For example, when the target information represents that the predicted interaction behavior information of the interactive object is a dangerous behavior, the predicted interaction information and the behavior of each interactive object that has occurred are displayed on a display screen or played by voice. For example, the interactive objects are a child and a pair of scissors, the interaction behavior is that the child picks up the scissors, and the behavior that has occurred is that the child stretches out his hand, then the text information of the child picking up the scissors and the child stretching out his hand is displayed on the display screen or played by voice, such as the child picking up the scissors and the child raising his hand.

[0048] In some embodiments, target information for the interactive objects is generated based on the predicted interactive behavior information and the actual behavior information of at least one of the interactive objects; it can be understood that the predicted interactive behavior information is verified based on the actual behavior information of at least one of the interactive objects, and if the predicted interactive behavior information can be realized through the actual behavior information, the type of the predicted interactive behavior information is determined, and if the type of the predicted interactive behavior information belongs to the target type, the target information is generated.

[0049] Exemplarily, the interactive objects are a child and a pair of scissors, the interactive behavior is the child picking up the scissors, and the behavior that has occurred is the child extending his hand, wherein the child picking up the scissors can be achieved by the child raising his hand, and it is further determined that the child picking up the scissors is a dangerous behavior and belongs to the target type of behavior, so that text information or image information of the child picking up the scissors and the child raising his hand is displayed on the display screen, or the child picking up the scissors and the child raising his hand are played through voice.

[0050] In an embodiment of the present application, at least two interactive objects in the first video are predicted by a trained interactive behavior prediction model to obtain predicted interactive behavior information of at least two interactive objects, and the predicted interactive information is determined based on the actual behavior information of at least one interactive object, thereby improving the accuracy of the predicted interactive behavior information, and when the predicted interactive behavior information is of a target type, target information for the interactive object is generated to display the target information on a display screen or play the target information through voice, thereby providing an early warning to the user to improve the user experience.

[0051] Figure 2 The following is a flow chart of an implementation of a behavior recognition method provided in an embodiment of the present application, which can be executed by a processor of an electronic device. Figure 1 , Figure 1 Step S102 in can be updated to step S201 to step S202, combining Figure 2 The steps shown are explained.

[0052] Step S201: Acquire position relationship information between at least two target interaction objects among the at least two interaction objects and status information of each target interaction object.

[0053] Wherein, the first target model includes a first model.

[0054] In some embodiments, the at least two target interaction objects represent at least two interaction objects that can interact; for example, the at least two interaction objects include a child, a scissors, a door handle, a chandelier, etc., wherein the at least two interaction objects that can interact include a child, a scissors, and a door handle, thereby determining the child, the scissors, and the door handle as the target interaction objects.

[0055] In some embodiments, the position relationship information between at least two target interactive objects may include the distance, angle, etc. between the target interactive objects, and the state information of each target interactive object may include the moving speed, etc. of the target interactive object.

[0056] Step S202: Generate the predicted interaction behavior information through the first model based on the position relationship information between the at least two target interaction objects and the state information of each target interaction object.

[0057] In some embodiments, the first model is a trained prediction model; used to predict at least two of the at least two interaction objects to obtain predicted interaction information; wherein the trained prediction model can be trained in the following manner:

[0058] Acquire multiple training samples, each training sample includes multiple video clips, each video clip includes at least two target interaction objects and annotation information for the at least two target interaction objects; and the category of at least one interaction behavior that has occurred between the at least two target interaction objects; wherein the annotation information includes the speed, movement direction, etc. of each target interaction object and the distance, orientation angle, etc. between the at least two target interaction objects; input the multiple video clips into the prediction model to be trained to obtain predicted interaction behavior information, adjust the model parameters of the prediction model to be trained based on the loss value between the predicted interaction behavior information and the category of at least one interaction behavior that has occurred between the at least two target interaction objects until the convergence condition is met, and output the trained prediction model.

[0059] In some embodiments, the positional relationship information between the at least two target interactive objects and the state information of each target interactive object are input into a trained prediction model to obtain at least one predicted interactive behavior information.

[0060] In an embodiment of the present application, at least two target interaction objects with which interactive behavior can occur are obtained from at least two interaction objects; and through a trained prediction model, based on the position relationship information of at least two target interaction pairs and the state information of each target interaction object, predicted interaction behavior information for at least two target interaction objects can be obtained.

[0061] In some embodiments, the first target model further includes a second model, the first video includes at least one video frame; and the method further includes:

[0062] By using the second model, at least two interactive objects in each video frame are screened to obtain at least two target interactive objects; the at least two target interactive objects include at least two interactive objects that can perform interactive behaviors.

[0063] In some embodiments, the second model is a trained screening model, which can be trained in the following manner: obtaining multiple samples; each sample includes multiple video data of interactive behaviors and corresponding annotation data, wherein each video frame of each video data is panorama segmented to obtain the category, size, and outline of each interactive object; the annotation data includes the category of interactive objects that can interact; multiple video data of interactive behaviors and corresponding annotation data are input into the screening model to be trained to obtain the target interactive object, and the model parameters of the screening model to be trained are adjusted based on the loss value between the target interactive object and the interactive object in the annotation data until the screening conditions are met, and the trained screening model is output.

[0064] In some embodiments, at least two interaction objects in each video frame in the first video are input into a trained screening model to obtain at least two target interaction objects that can interact.

[0065] In an embodiment of the present application, at least two interaction objects in each video frame of the first video are screened by a trained screening model to obtain at least two target interaction objects that can interact, thereby improving the accuracy of subsequent predictions.

[0066] In some embodiments, the behaviors of the at least two interactive objects may be implemented in the following ways:

[0067] Based on the second target model and the first video, information about behaviors that have occurred for the at least two interacting objects is generated.

[0068] In some embodiments, the second target model is a trained image recognition model; the trained image recognition model can be trained in the following manner: obtain multiple training samples, each training sample includes at least one object in each of a plurality of consecutive video frames; and the behavior that has occurred for each object; input at least one object in each of a plurality of consecutive video frames into the image recognition model to be trained, obtain the predicted behavior that has occurred for each object, adjust the model parameters of the image recognition model to be trained based on the loss value between the predicted behavior that has occurred and the behavior that has occurred for each object, until the loss value between the predicted behavior that has occurred and the behavior that has occurred for each object meets the convergence condition, and output the trained image recognition model.

[0069] In some embodiments, at least two interacting objects in each video frame of the first video are input into a trained image recognition model to obtain information about behaviors that have occurred of the at least two interacting objects; wherein the information about behaviors that have occurred of the at least two interacting objects may be the behaviors that have occurred of each interacting object itself, or may be the interactive behaviors between two interacting objects.

[0070] In an embodiment of the present application, information on past behaviors of at least two interacting objects in a video frame in a first video is obtained through a trained image recognition model, so that the predicted interaction behavior information is verified through the information on past behaviors of at least two interacting objects, thereby improving the accuracy of the prediction and improving the user experience.

[0071] Figure 3 The following is a flow chart of an implementation of a behavior recognition method provided in an embodiment of the present application, which can be executed by a processor of an electronic device. Figure 1 , Figure 1 Step S103 in can be updated to step S301 to step S302, combining Figure 3 The steps shown are explained.

[0072] Step S301: Generate a matching result based on the predicted interaction behavior information and the actual behavior information of at least one of the interaction objects.

[0073] In some embodiments, the predicted interaction behavior information is obtained by predicting the interaction behavior of at least two interaction objects in the current frame of the first video and the multiple consecutive frames before the current frame; the past behavior information of at least one of the interaction objects is also obtained by identifying at least one of the interaction objects in the current frame of the first video and the multiple consecutive frames before the current frame, thereby verifying the predicted interaction behavior information of at least two interaction objects in the current frame and the multiple consecutive frames before the current frame based on the past behavior of at least one of the interaction objects in the current frame and the multiple consecutive frames before the current frame.

[0074] In other embodiments, it can be understood that the predicted interaction behavior information can be obtained by predicting the interaction behavior of at least two interactive objects in the current frame of the first video and the multiple consecutive frames before the current frame; the past behavior information of at least one of the interactive objects can be obtained by identifying the multiple consecutive frames of the current frame at the next moment; thereby, based on the past behavior information of at least one of the interactive objects in the current frame and the multiple consecutive frames after the current frame, the predicted interaction behavior information of at least two interactive objects in the current frame and the multiple consecutive frames before the current frame is verified.

[0075] In some embodiments, the behavior of at least one of the interactive objects may be a behavior initiated by the interactive object itself; or it may be an interactive behavior in which a dynamic interactive object actively contacts a stationary interactive object or a dynamic interactive object actively contacts another dynamic interactive object.

[0076] Exemplarily, the interactive objects include a first person, a table, and a second person, then at least one of the interactive objects' occurred behaviors may be a person touching the table, the first person or the second person extending a hand, the first person actively contacting the second person, etc.

[0077] In some embodiments, the matching result may include a situation where the actual behavior information of at least one of the interactive objects matches the predicted interactive behavior information; it is understandable that the interactive object can implement the interactive behavior corresponding to the predicted interactive behavior information based on the actual behavior information.

[0078] Exemplarily, at least two interactive objects are a child and a pair of scissors, and the predicted interactive behavior information is described by the child picking up the scissors and the child stretching out his hand. Among them, by verifying the child picking up the scissors through the child stretching out his hand, it can be seen that the child can realize the interactive behavior of picking up the scissors by stretching out his hand, which indicates that the child stretching out his hand and the child picking up the scissors are matched.

[0079] In some embodiments, the matching result may also include a situation where the actual behavior information of at least one of the interactive objects does not match the predicted interactive behavior information; it is understandable that the interactive object cannot implement the predicted interactive behavior information based on the actual behavior information.

[0080] Exemplarily, take the interactive objects of a baby, a stroller, and a door handle as an example; wherein the baby's existing behavior information is that the baby lies in the stroller and reaches out his hand, and the predicted interactive behavior information is that the baby touches the door handle; it is understandable that the baby lying in the stroller cannot realize the interactive behavior of touching the door handle, thereby representing that the baby lying in the stroller and reaching out his hand does not match the baby touching the door handle.

[0081] In other embodiments, the matching result may also indicate that the predicted interaction behavior information is inaccurate; it is understandable that the predicted interaction behavior information conflicts with the interaction behavior that has occurred or real common sense.

[0082] For example, take the interactive objects as a baby, a stroller, and a table; the information on the actual behavior is that the baby is lying in the stroller, and the predicted interactive behavior information is that the baby walks toward the table. It can be seen that the predicted interactive behavior information conflicts with the information on the actual behavior and common sense in reality, thereby determining that the predicted interactive behavior information is inaccurate.

[0083] Step S302: when the matching result indicates that the interactive object can realize the predicted interactive behavior information based on the occurred behavior information, generate target information for the interactive object.

[0084] In some embodiments, when the matching result represents that the interactive object can realize the predicted interactive behavior information based on the behavior information that has occurred, the interactive behavior type of the predicted interactive behavior information of the interactive object is determined, and when the interactive behavior type is a target type, target information for the interactive object is generated; wherein the target information may be warning information, prompt information, etc. for the predicted interactive behavior information of the interactive object.

[0085] Exemplarily, the interactive objects are a child and a pair of scissors; the behavior information that has occurred is that the child extends his hand, and the predicted interactive behavior information is that the child takes the scissors. It can be seen that the predicted interactive behavior information is a dangerous interactive behavior, which belongs to a target type of interactive behavior, thereby generating target information for the child taking the scissors; the target information can be a text message or voice message that the child is extending his hand and may be about to take the scissors, and the text message that the child is extending his hand and may be about to take the scissors is displayed on the display screen or the voice message is broadcast by voice.

[0086] In an embodiment of the present application, the predicted interactive behavior information is verified by the information on the behavior that has occurred to obtain matching information characterizing whether the predicted interactive behavior information and the information on the behavior that has occurred match, so as to improve the accuracy of the predicted interactive behavior information; when the predicted interactive behavior information and the information on the behavior that has occurred match, target information for the interactive object is generated, and the target information is displayed on a display screen or played by voice to prompt the user of the upcoming interactive behavior of the interactive object, thereby improving the user experience.

[0087] Figure 4 The following is a flow chart of an implementation of a behavior recognition method provided in an embodiment of the present application, which can be executed by a processor of an electronic device. Figure 3 , Figure 3 Step S302 in can be updated to step S401, combining Figure 4 The steps shown are explained.

[0088] Step S401: When the matching result represents that the interactive object can realize the predicted interactive behavior information based on the behavior information that has occurred, determine whether the predicted interactive behavior information is an interactive behavior of the target type; if the predicted interactive behavior information meets the interactive behavior of the target type, generate target information for the interactive object.

[0089] In some embodiments, it can be understood that when the interactive behavior type is a target type, target information is generated for the interactive object; wherein the target information can be warning information, prompt information, etc. of the predicted interactive behavior information of the interactive object, and the target information can be displayed on a display screen or played through voice.

[0090] Exemplarily, the interactive objects are a child and a pair of scissors; the behavior information that has occurred is that the child extends his hand, and the predicted interactive behavior information is that the child takes the scissors. It can be seen that the predicted interactive behavior information is a dangerous interactive behavior, which belongs to a target type of interactive behavior, thereby generating target information for the child taking the scissors; the target information can be a text message or voice message that the child is extending his hand and may be about to take the scissors, and the text message that the child is extending his hand and may be about to take the scissors is displayed on the display screen or the voice message is broadcast by voice.

[0091] In some embodiments, if the predicted interaction behavior information does not match the target type of interaction behavior, the current recognition and prediction process is terminated. For example, the interaction objects are a child and a toy, where the behavior information that has occurred is the child reaching out his hand, and the predicted interaction behavior information is the child taking the toy, and the child taking the toy does not belong to the target type, the current recognition and prediction process is terminated.

[0092] In an embodiment of the present application, when the predicted interactive behavior of the interactive object is of a target type, the target information is displayed on a display screen or played by voice to prompt the user of the upcoming interactive behavior of the interactive object, thereby improving the user experience.

[0093] Figure 5 The following is a flow chart of an implementation of a behavior recognition method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The process may include step S501 and step S502. Figure 5 The steps shown are explained.

[0094] Step S501: Based on a user's selection instruction for a target object among at least two objects in a first video, at least two objects in a video frame in a first video are identified to obtain at least two target objects and a category of each of the target objects.

[0095] In some embodiments, the first video includes at least one video frame, each video frame including at least two objects.

[0096] In some embodiments, the selection instruction may be a selection instruction input by the user through voice, text, touch, or a selection instruction input by framing a target area in an image.

[0097] In some embodiments, based on a selection operation input by a user through voice, text, touch, box selection, etc., a target object among at least two objects in each video frame is identified to obtain at least two target objects and a category of each target object.

[0098] Exemplarily, at least two objects in the video frame include a first child, a second child, an adult, scissors, a window, a wall, the ground, etc. Based on the user's selection operation input through voice, text, touch, box selection, etc., at least two target objects, the first child, the second child, the scissors, the window, the wall, and the ground, are determined.

[0099] Step S502: Based on the category of each of the target objects, the at least two target objects are screened to obtain at least two interactive objects.

[0100] In some embodiments, based on the category of each of the target objects, target objects of a specific category among the at least two target objects are screened to obtain at least two interactive objects; wherein the target objects of the specific category may be target objects representing a background category.

[0101] Exemplarily, at least two target objects are a first child, a second child, scissors, a window, a wall, and the ground. Based on the category of each target object, the target objects of the background category are filtered to obtain at least two interactive objects, namely the first child, the second child, the scissors, and the window.

[0102] In an embodiment of the present application, the original first video is identified based on the user's selection instructions to obtain at least two target objects that the user needs to identify, and based on the category of each target object, the target objects belonging to a specific category among the at least two target objects are eliminated to obtain at least two interactive objects that can interact, thereby improving the accuracy of subsequent predictions to improve the user experience.

[0103] The following describes an exemplary application of a behavior recognition method provided in an embodiment of the present application in an actual scenario.

[0104] Figure 6A schematic diagram of an implementation method of a person behavior recognition method provided for an embodiment of the present application includes a continuous video sequence 601 and a behavior recognition model 602; wherein the continuous video sequence 601 includes multiple continuous video frames; each video frame includes a target person; the behavior recognition model 602 is used to extract frame-by-frame features of the target person in each video frame and fuse time domain changes, and output the specific behavior category of the target person; specifically, by calculating independent features of different frame images and mutually correlated features within a time length T, the behavior of the target person is judged to obtain the specific behavior category of the target person.

[0105] Figure 7 A schematic diagram of an implementation method of a person behavior recognition method provided for an embodiment of the present application includes a continuous video sequence 701, a preprocessing module 702, and a behavior recognition model 703; wherein the continuous video sequence 701 includes multiple continuous video frames; each video frame includes a target person; the preprocessing module 702 is used to extract a human skeleton point sequence for the target person in each video frame; the behavior recognition model 703 is used to extract features from each video frame including the human skeleton point sequence of the target person and fuse the time domain changes to obtain the specific behavior category of the target person. It can be understood that the person behavior category is determined by the morphological changes of the skeleton points within time T.

[0106] However, whether the input is the original image or the pre-processed method, it can only judge the actions that have already occurred, and cannot warn about the actions that have not yet occurred. Moreover, it relies on a single video form input, which has limited effect when processing the interaction between people and objects. (For example, when a hand uses scissors or clicks a mouse, it is necessary to first identify objects of different categories, and then combine them with the human behavior to derive the specific category.) There are many categories of objects that can interact with people, but the categories that can be determined by image features are limited, and the behavior recognition results ultimately depend to a large extent on the accuracy of the object recognition results, although some human body movements are similar or even the same (for example, if scissors are mistakenly identified as a mouse, it will not trigger a danger warning).

[0107] Multimodal large language models have been widely used in logical reasoning. They can describe and understand the content contained in an image. However, large language models cannot directly and quickly output recognition results for specific target areas. For example, if there are multiple people in the picture and the user only cares about one of them, it is necessary to give the large language model prompts and guidance through multiple rounds of interaction so that it can understand the area that the user needs to pay attention to. Figure 8 , Figure 8 A schematic diagram of a video frame provided for an embodiment of the present application, which includes a first area 801, a second area 802, and a third area 803; wherein the person in the third area 803 is the target person that the user needs to pay attention to.

[0108] Fig. 9 The following is a flow chart of an implementation of a behavior recognition method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The process may include steps S901 to S905. Fig. 9 The steps shown are explained.

[0109] Step S901: Detect the persons and objects contained in each video frame in the input video sequence (corresponding to the first video in the above embodiment) and retain the target persons and target objects (corresponding to at least two interactive objects in the above embodiment).

[0110] In some embodiments, panoramic segmentation is performed on each video frame in the input video sequence to obtain the outline and category of each object and person; obtain the target person and target object that need to be paid attention to; and eliminate objects in specific backgrounds such as walls and ground to reduce subsequent interference.

[0111] Step S902: input the target person and the target object into the pre-trained large model for predicting human-object interaction (corresponding to the second model in the above embodiment) to obtain the interacting persons and interacting objects (corresponding to at least two target interacting objects in the above embodiment) with which interaction may occur.

[0112] Among them, the pre-trained large model for judging human-object interaction can be trained by the following method: obtaining multiple videos of interactive behaviors, performing panoramic segmentation on a preset number of video frames before the video frame in which the interactive behaviors have occurred, and obtaining the outlines and categories of all objects and all characters in each frame; marking the category of each object, the size of each object, and the distance between the object and the character; and using the objects that have not interacted with the characters as the background; inputting multiple videos of interactive behaviors into the large model for predicting human-object interaction to be trained, and obtaining predicted interactive objects and interactive persons, and adjusting the model parameters of the large model for predicting human-object interaction to be trained based on the predicted interactive objects and interactive persons and the loss values ​​of the interactive persons and interactive objects in the interactive behavior videos until the convergence conditions are met, and outputting the pre-trained large model for predicting human-object interaction.

[0113] Fig.10 The present invention provides a method for determining an interactive object and an interactive person according to an embodiment of the present invention. The method can be executed by a processor of an electronic device. The process may include steps S1001 to S1004. Fig.10 The steps shown are explained.

[0114] Step S1001: input the target object and target person.

[0115] Step S1002: make a judgment based on the pre-trained human-object interaction judgment model.

[0116] Step S1003: Whether there will be interaction between the target object and the target person.

[0117] Step S1004: If interaction is likely to occur, the target object and the target person are determined as the interactive object and the interactive person.

[0118] Among them, if no interaction will occur, the target object and the target person will be deleted.

[0119] In some embodiments, the pre-trained person-object interaction prediction model obtains the interacting objects and interacting persons based on the category, size, and distance between each target object and the target person.

[0120] Step S903: input the interactive object and the interactive person into the pre-trained large model for judging the potential behavior of the person (corresponding to the first model in the above embodiment) to obtain the predicted interactive behavior between the interactive person and the interactive object (corresponding to the predicted interactive behavior information in the above embodiment).

[0121] Fig.11 The following is a schematic diagram of an implementation flow of an interactive behavior prediction method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The process may include steps S1101 to S1103. Fig.11 The steps shown are explained.

[0122] Step S1101: input the interactive objects and interactive characters into the pre-trained large model for judging potential behaviors of personnel.

[0123] Step S1102: Prediction is performed based on the input into the pre-trained personnel potential behavior judgment model.

[0124] Step S1103: output the predicted interactive behaviors of the interactive objects and the interactive characters.

[0125] Step S904: based on the image model and the input video sequence, obtain the behaviors that have occurred of the interacting personnel (corresponding to the behavior information that has occurred in the above embodiment).

[0126] Step S905: If the actual behavior of the interacting person is consistent with the predicted interaction behavior, an early warning is triggered.

[0127] In some embodiments, when the actual behavior of the interacting person is consistent with the predicted interaction behavior between the interacting person and the interactive object, if the predicted interaction behavior between the interacting person and the interactive object is a dangerous behavior, an early warning is triggered.

[0128] Fig.12 A schematic diagram of a prediction result provided in an embodiment of the present application includes text information of behaviors that have occurred and text information 1202 of predicted interactive behaviors, and is displayed on a display screen 1201 .

[0129] In an embodiment of the present application, while considering the pixel-level detail information of the image (attribute information such as the categories of people and objects in the image), the description information of the video image (objects, character interactions or actions) of the large language model is integrated to enable the model to fully combine inputs of different forms and improve the quality of the final output results.

[0130] Based on the foregoing embodiments, the embodiments of the present application provide a behavior recognition device, which includes the units included and the modules included in the units, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0131] Fig.13 A schematic diagram of the structure of a behavior recognition device provided in an embodiment of the present application is shown in FIG. Fig.13 As shown, the behavior recognition device 1300 includes: an acquisition module 1301, a first generation module 1302, and a second generation module 1303, wherein: the acquisition module 1301 is used to obtain a first video; the first video includes at least two interactive objects; the first generation module 1302 is used to generate predicted interactive behavior information between the at least two interactive objects based on a first target model and the first video; the second generation module 1303 is used to generate target information for the interactive objects based on the predicted interactive behavior information and at least one of the interactive objects' actual behavior information, and the target information can be displayed on a display screen or played by voice; the actual behavior information is obtained based on the first video.

[0132] In some embodiments, the first target model includes a first model; the first generation module 1302 is also used to obtain the position relationship information between at least two target interaction objects among the at least two interaction objects and the status information of each target interaction object; through the first model, the predicted interaction behavior information is generated based on the position relationship information between the at least two target interaction objects and the status information of each target interaction object.

[0133] In some embodiments, the first target model also includes a second model; the first video includes at least one video frame; the behavior recognition device 1300 also includes a screening module (not shown in the figure), which is used to screen at least two interactive objects in each video frame through the second model to obtain at least two target interactive objects; the at least two target interactive objects include at least two interactive objects that can perform interactive behaviors.

[0134] In some embodiments, the first generating module 1302 is further used to generate information about behaviors that have occurred for the at least two interactive objects based on the second target model and the first video.

[0135] In some embodiments, the second generation module 1303 is further used to generate a matching result based on the predicted interaction behavior information and the actual behavior information of each of the interaction objects; and to generate target information for the interaction object when the matching result represents that the interaction object can realize the predicted interaction behavior information based on the actual behavior information.

[0136] In some embodiments, the second generation module 1303 is also used to determine whether the predicted interaction behavior information is an interaction behavior of the target type when the matching result represents that the interaction object can realize the predicted interaction behavior information based on the behavior information that has occurred, and if the predicted interaction behavior information is consistent with the target type of interaction behavior, generate target information for the interaction object.

[0137] In some embodiments, the behavior recognition device 1300 also includes a preprocessing module (not shown in the figure), which is used to identify at least two objects in the video frame of the first video based on the user's selection instruction for the target object among the at least two objects in the first video, and obtain at least two target objects and the category of each of the target objects; based on the category of each of the target objects, the at least two target objects are screened to obtain at least two interactive objects.

[0138] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0139] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0140] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0141] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0142] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in an electronic device, a processor in the electronic device executes some or all of the steps for implementing the above method.

[0143] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0144] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0145] Fig.14 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Fig.14 As shown, the hardware entity of the electronic device 1400 includes: a processor 1401 and a memory 1402, wherein the memory 1402 stores a computer program that can be run on the processor 1401, and the processor 1401 implements the steps in the method of any of the above embodiments when executing the program.

[0146] The memory 1402 stores computer programs that can be run on the processor. The memory 1402 is configured to store instructions and applications executable by the processor 1401. It can also cache data to be processed or processed by the processor 1401 and various modules in the electronic device 1400 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0147] When the processor 1401 executes the program, the steps of any of the above methods are implemented. The processor 1401 generally controls the overall operation of the electronic device 1400.

[0148] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the method of any of the above embodiments.

[0149] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0150] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0151] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0152] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0153] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0156] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0157] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0158] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A behavior recognition method, the method comprising: Get the first video; The first video includes at least two interactive objects; Based on the first target model and the first video, generating predicted interaction behavior information between the at least two interaction objects; Based on the predicted interaction behavior information and the behavior information of at least one of the interaction objects, generating target information for the interaction object, wherein the target information can be displayed on a display screen or played by voice; The occurred behavior information is obtained based on the first video.

2. According to the method of claim 1, the first target model comprises a first model; and the generating, based on the first target model and the first video, predicted interaction behavior information between the at least two interaction objects comprises: Acquire positional relationship information between at least two target interactive objects among the at least two interactive objects and status information of each target interactive object; The predicted interaction behavior information is generated through the first model based on the position relationship information between the at least two target interaction objects and the state information of each target interaction object.

3. The method according to claim 2, wherein the first target model further comprises a second model; the first video comprises at least one video frame; and the method further comprises: Using the second model, at least two interaction objects in each video frame are screened to obtain at least two target interaction objects; The at least two target interactive objects include at least two interactive objects that can perform interactive behaviors.

4. The method according to claim 1, further comprising: Based on the second target model and the first video, information about behaviors that have occurred for the at least two interacting objects is generated.

5. The method according to any one of claims 1 to 4, wherein generating target information for the interactive object based on the predicted interactive behavior information and at least one of the interactive objects' actual behavior information comprises: Generate a matching result based on the predicted interaction behavior information and the actual behavior information of each interaction object; In a case where the matching result represents that the interactive object can realize the predicted interactive behavior information based on the occurred behavior information, target information for the interactive object is generated.

6. The method according to claim 5, wherein, when the matching result indicates that the interactive object can realize the predicted interactive behavior information based on the occurred behavior information, generating target information for the interactive object comprises: When the matching result represents that the interactive object can realize the predicted interactive behavior information based on the behavior information that has occurred, it is determined whether the predicted interactive behavior information is an interactive behavior of the target type; if the predicted interactive behavior information meets the target type of interactive behavior, target information for the interactive object is generated.

7. The method according to any one of claims 1 to 4, further comprising: Based on a user's selection instruction for a target object among at least two objects in the first video, identifying at least two objects in a video frame in the first video, and obtaining at least two target objects and a category of each of the target objects; Based on the category of each of the target objects, the at least two target objects are screened to obtain at least two interactive objects.

8. A behavior recognition device, comprising: An acquisition module, used for acquiring a first video; The first video includes at least two interactive objects; A first generating module, configured to generate predicted interaction behavior information between the at least two interaction objects based on a first target model and the first video; A second generating module, configured to generate target information for the interactive object based on the predicted interactive behavior information and the information of the at least one interactive object's existing behavior, wherein the target information can be displayed on a display screen or played by voice; The occurred behavior information is obtained based on the first video.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images