Behavior event recognition method, device, equipment and computer storage medium

Abstract

The present application discloses a method, apparatus, device and computer storage medium for identifying behavioral events, the method comprising: obtaining multiple video data of the same conference scene shot by multiple shooting devices at the same time from different angles; performing object recognition on the video clips of the target time period in each video data respectively, and obtaining the seat coordinates of the seat where the person object is located in the target area; determining the seat change information of the person object based on the seat coordinates of the initial seat and the target seat; tracking the person object in the video clip to obtain the motion trajectory information of the person object; fusing the motion trajectory information and the seat change information to determine the behavioral event of the person object. According to the embodiment of the present application, the visual features of the person object can be more comprehensively reflected, and the motion trajectory information and the seat change information can be fused, thereby improving the accuracy of behavioral event recognition while solving the information fusion problem.

Description

Technical Field

[0001] The present application belongs to the field of image recognition technology, and in particular relates to a behavior event recognition method, device, equipment and computer storage medium. Background Art

[0002] At present, traditional target detection and behavior analysis methods are mostly based on monocular camera scenes, mainly focusing on the basic behavior of the target object in the scene, and usually do not involve intention behavior recognition and behavior category analysis. In scenes with many target objects and more complex behaviors, binocular cameras can be used to obtain visual features of different dimensions such as the front, side or back of the target object, thereby identifying the behavior category of the target object.

[0003] However, due to the limitations of camera angles and shooting distances at different locations, the visual features of the target object are not clear enough, which affects the accuracy of target object recognition and behavior analysis, and further increases the difficulty of identifying the target object's behavioral events. Therefore, how to more accurately identify the target object's behavioral events is one of the challenges faced by researchers in this field. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, device, and computer storage medium for behavioral event recognition, which can more comprehensively reflect the visual characteristics of human objects, fuse motion trajectory information and seat change information, avoid the recognition impact caused by occlusion or image superposition in the video, and improve the accuracy of behavioral event recognition while solving information fusion.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying behavioral events, the method comprising:

[0006] Acquire multiple video data of the same conference scene shot by multiple shooting devices at the same time from different angles;

[0007] Perform object recognition on the video clips of the target time period in each video data to obtain the seat coordinates of the seat where the person object is located in the target area, where the seat includes the initial seat and the target seat;

[0008] Determine seat change information of the person object based on the seat coordinates of the initial seat and the seat coordinates of the target seat;

[0009] Tracking the human objects in the video clips to obtain the motion trajectory information of the human objects;

[0010] The motion trajectory information and seat change information are integrated to determine the behavioral events of the character object.

[0011] In some implementations of the first aspect, the video data includes multiple video frames, and object recognition is performed on each video segment of the target time period in the video data to obtain the seat coordinates of the seat where the human object is located in the target area, including:

[0012] Using the target detection model to identify the image of the target area in the video frame, determine the seat coordinates of the seat where the person object is sitting in the target area and the coordinates of the position where the person object and the seat object overlap;

[0013] Determine the overlapping area of ​​the person object and the seat based on the seat coordinates of the seat where the person object is seated and the coordinates of the overlapping position between the person object and the seat where the person object is seated;

[0014] The seat where the person object is seated is determined based on the overlapping area between the person object and the seat where the person object is seated.

[0015] In some implementations of the first aspect, if the identification result of the seat left is not unique, the method further includes:

[0016] Obtain a first overlapping area between the person object and the first seat, and a second overlapping area between the person object and the second seat, respectively. The first seat and the second seat are adjacent seats.

[0017] Comparing the first overlapping area, the second overlapping area, and a preset overlapping area threshold;

[0018] When the first overlapping area is greater than the second overlapping area and the preset overlapping area threshold, the first seat is determined as a seat that has not been occupied;

[0019] When the second overlapping area is greater than both the first overlapping area and the preset overlapping area threshold, the second seat is determined as a seat that has not been occupied.

[0020] In some implementations of the first aspect, tracking a person object in a video clip to obtain motion trajectory information of the person object includes:

[0021] During a second target period of the conference scene, the target tracking model is used to track the human object, and target point coordinates of the human object in different video frames during the second target period are determined;

[0022] Based on the target point coordinates of the human object in each video frame within the second target time period, motion trajectory information of the human object is obtained.

[0023] In some implementations of the first aspect, before obtaining the motion trajectory information of the human object based on the target point coordinates of the human object in each video frame within the second target time period, the method further includes:

[0024] encoding the body features of the human object in each video frame within the second target time period using a feature encoding algorithm;

[0025] Calculating the encoding distance of the same human body feature in different video frames within the second target time period;

[0026] Comparing the encoded distance with a preset threshold to determine whether the human objects in different video frames are the same human object;

[0027] When the person objects in different video frames are the same person object, the motion trajectory information of the person object is determined.

[0028] In some implementations of the first aspect, when there are multiple behavior events, the method further includes:

[0029] Determine the starting time of each two adjacent behavioral events;

[0030] When the time difference between the start times of two adjacent behavior events is less than a preset value, the two adjacent behavior events are merged into one behavior event.

[0031] In some implementations of the first aspect, fusing motion trajectory information and seat change information to determine a behavior event of a person object includes:

[0032] The motion trajectory information and seat change information are weighted and fused according to preset weights to determine the behavioral events of the character objects.

[0033] In a second aspect, an embodiment of the present application provides a behavior event recognition device, the device comprising:

[0034] An acquisition module is used to acquire multiple video data of the same conference scene shot by multiple shooting devices at the same time from different angles;

[0035] an identification module, configured to perform object identification on each video segment of a target time period in the video data, and obtain the seat coordinates of a seat where a person object is located in a target area, wherein the seat where a person is located includes an initial seat where a person is located and a target seat where a person is located;

[0036] A first determining module is configured to determine seat change information of the person object based on the seat coordinates of the initial seat and the seat coordinates of the target seat;

[0037] Tracking module, used to track the human object in the video clip and obtain the motion trajectory information of the human object;

[0038] The second determination module is used to fuse the motion trajectory information and the seat change information to determine the behavior event of the character object.

[0039] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0040] When the processor executes the computer program instructions, the behavior event recognition method as described in any one of the first aspects is implemented.

[0041] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the behavioral event recognition method as described in any one of the first aspects is implemented.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device can execute the behavioral event recognition method described in any one of the first aspects.

[0043] The behavioral event recognition method, device, equipment and computer storage medium of the embodiment of the present application obtains multiple video data of the same conference scene shot by multiple shooting devices at the same time from different angles, and performs object recognition on the video clips of the target time period in each video data respectively, and obtains the seat coordinates of the seat where the person object is located in the target area. The seat where the person object is sitting includes the initial seat where the person object is sitting and the target seat where the person object is sitting. Then, based on the seat coordinates of the initial seat where the person object is sitting and the target seat where the person object is sitting, the seat change information of the person object is determined, and the person object in the video clip is tracked to obtain the motion trajectory information of the person object, thereby fusing the motion trajectory information and the seat change information to determine the behavioral event of the person object. In this process, the video data is shot from different angles and can more comprehensively reflect the visual features of the person object. When determining the behavioral event of the person object, the motion trajectory information and the seat change information are fused, which avoids the recognition influence caused by occlusion or image superposition in the video, and improves the accuracy of behavioral event recognition while solving the information fusion problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 This is a flow chart of a behavior event recognition method provided in an embodiment of the present application;

[0046] Figure 2 is a flowchart of a specific implementation method of step S102;

[0047] Figure 3This is a flow chart of a method for determining initial seats provided in an embodiment of the present application;

[0048] Figure 4 is a schematic diagram of overlapping seats provided in an embodiment of the present application;

[0049] Figure 5 is a flowchart of a specific implementation method of step S104;

[0050] Figure 6 This is a flow chart of a method for identifying a person object provided in an embodiment of the present application;

[0051] Figure 7 This is a flow chart of a behavior event merging method provided in an embodiment of the present application;

[0052] Figure 8 This is a schematic diagram of the structure of a behavior event recognition device provided in an embodiment of the present application;

[0053] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0055] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0056] Currently, traditional target detection and behavior analysis methods are mostly based on monocular camera scenarios, focusing on the target object's basic behavior in the scene. They usually do not involve intentional behavior recognition or behavior category analysis. In binocular camera scenarios, due to the cross-border camera, the complexity of the binocular scene is increased. Occlusion or image overlap is often present in the video, which poses a huge challenge to information fusion and calibration technologies. For example, due to the limitations of the shooting angle and shooting distance of two cameras in different positions, the visual features of different dimensions such as the front, side, and back of the person in the video are often not clear enough, making image matching more difficult and directly affecting the fusion of person information.

[0057] In order to solve the problems of the prior art, the embodiments of the present application provide a behavior event recognition method, apparatus, device and computer storage medium.

[0058] Figure 1 FIG. 1 is a flow chart of a behavior event recognition method provided by an embodiment of the present application. Figure 1 As shown, the method specifically includes the following steps:

[0059] S101, obtaining multiple video data of the same conference scene shot from different angles at the same time;

[0060] S102, performing object recognition on each video segment of the target time period in each video data to obtain the seat coordinates of the seat where the person object is located in the target area, where the seat where the person object is located includes the initial seat where the person object is located and the target seat where the person object is located;

[0061] S103, determining seat change information of the person object based on the seat coordinates of the initial seat and the seat coordinates of the target seat;

[0062] S104, tracking the human object in the video clip to obtain motion trajectory information of the human object;

[0063] S105: Fusing the motion trajectory information and the seat change information to determine the behavior event of the character object.

[0064] Therefore, according to the behavioral event recognition method provided by the embodiment of the present application, multiple video data of the same conference scene shot by multiple shooting devices at the same time from different angles are obtained, and object recognition is performed on the video clips of the target time period in each video data respectively, and the seat coordinates of the seat where the person object is located in the target area are obtained. The seat where the person object is sitting includes the initial seat where the person object is sitting and the target seat where the person object is sitting. Then, based on the seat coordinates of the initial seat where the person object is sitting and the target seat where the person object is sitting, the seat change information of the person object is determined, and then the person object in the video clip is tracked to obtain the motion trajectory information of the person object, thereby fusing the motion trajectory information and the seat change information to determine the behavioral event of the person object. In this process, the video data is shot from different angles and can more comprehensively reflect the visual features of the person object. When determining the behavioral event of the person object, the motion trajectory information and the seat change information are fused, avoiding the recognition influence caused by occlusion or image superposition in the video, while solving the information fusion problem, the accuracy of behavioral event recognition is improved.

[0065] In some embodiments, in S101, the video data may be obtained by capturing the same conference scene at the same time using a diagonally mounted binocular camera in an upside-down overhead mode. Due to limitations in camera mounting angles, the features of the human subjects captured in the two video data sets, such as those captured by the diagonally mounted cameras, may be different.

[0066] In some embodiments, in S102, based on the experience of the conference consensus, a conference video can be divided into a preparation period, a drop-off period, an ongoing period, and an ending period. For example, the period before the meeting starts is defaulted to the preparation stage, the period when the participants leave is defaulted to the ending stage, a certain period after the meeting starts is set as the drop-off period, for example, about three minutes, and the period from three minutes after the start of the meeting to the end stage is set as the ongoing period. Among them, the target period refers to the drop-off period. In combination with the conference scene, the objects in the video clip may include people or physical objects such as tables and chairs, and the target area refers to the area demarcated by the area where the conference tables and chairs are located, so as to perform object recognition on the video clip, thereby obtaining the seat coordinates of the drop-off seats of the people objects in the target area, wherein the drop-off seats include the initial drop-off seats and the target drop-off seats.

[0067] As an example, a video clip includes multiple video frames, such as Figure 2 As shown, the above S102 may specifically include the following steps:

[0068] S1021: using a target detection model to identify an image of a target area in a video frame, and determining the seat coordinates of a seat where a person object is seated in the target area and the coordinates of a position where the person object and the seat are overlapped;

[0069] S1022: Determine the overlapping area of ​​the person object and the seat where the person object is seated based on the seat coordinates of the seat where the person object is seated and the coordinates of the overlapping position between the person object and the seat where the person object is seated;

[0070] S1023: Determine the seat where the person object is seated based on the overlapping area between the person object and the seat where the person object is seated.

[0071] For example, in S1031, the target detection model can be a YOLOv5 model, which mainly performs target detection based on a convolutional neural network (CNN), thereby obtaining the seat coordinates of the seat where the person object is seated in the target area and the coordinates of the overlapping position between the person object and the seat where the person object is seated. There are multiple seats in the target area, and the seat where the person object is seated refers to the seat where the person object is seated. In S1032, due to the influence of the shooting angle, the recognition result of the seat where the person object is seated may not be unique, so it is necessary to determine the overlapping area between the person object and the seat where the person object is seated, and then determine the seat where the person object is seated.

[0072] For example, in S1033, if the identification result of the seat is not unique, in order to determine the initial seat of the person object, such as Figure 3 As shown, the method may further include the following steps:

[0073] S301, respectively obtaining a first overlapping area between the person object and a first seat, and a second overlapping area between the person object and a second seat, where the first seat and the second seat are adjacent seats;

[0074] S302: Compare the first overlapping area, the second overlapping area, and a preset overlapping area threshold;

[0075] S303: If the first overlapping area is greater than both the second overlapping area and the preset overlapping area threshold, the first seat is determined as a seat that has been left unoccupied.

[0076] S304: When the second overlapping area is greater than both the first overlapping area and the preset overlapping area threshold, determine the second seat as a seat that has been taken.

[0077] As an example, Figure 4 As shown, the first area represents the first overlapping area between the person object and the first seat, and the second area represents the second overlapping area between the person object and the second seat. For example, with 1 representing the whole, the preset overlapping area threshold can be set to 0.75, which means that if the overlapping area between the person object and the seat is greater than 75% of the seat area, the seat can be considered as the seat where the person object sat. If the identification result of the seat where the person object sat is not unique, the seat where the person object sat is determined by comparing the first overlapping area, the second overlapping area, and the preset overlapping area threshold.

[0078] Therefore, by setting an overlapping area threshold, and when the seat identification result is not unique, comparing the first overlapping area between the person object and the first seat, the second overlapping area between the person object and the second seat, and the size of the preset overlapping area threshold, the seat where the person object is seated can be accurately determined.

[0079] In some embodiments, in S103, the initial seat refers to the original seat of the person object, and the target seat refers to the seat of the person object after the motion trajectory changes. By determining the seat coordinates of the initial seat and the target seat, the seat change information of the person object can be obtained.

[0080] In some embodiments, in S104, during the process of tracking a person object, the person object being tracked may leave the camera shooting range, or a new person object may enter the camera shooting range, or in the video shot by a camera installed at a diagonal angle, the person object can be identified in the video data shot at the first angle, but cannot be identified in the video data shot at the second angle. Therefore, this process involves cross-border behavior tracking of the person object, and cross-border refers to the person object entering or leaving the camera shooting range. As an example, Figure 5 As shown, the above S104 may specifically include the following steps:

[0081] S1041. Tracking a human object using a target tracking model during a second target period of the conference scene to determine target point coordinates of the human object in different video frames during the second target period.

[0082] S1042: Obtain motion trajectory information of the human object based on the target point coordinates of the human object in each video frame within the second target time period.

[0083] For example, in S1041, the second target period refers to the period of the meeting, and the target point is a detection point that can reflect the behavioral actions of the character object, such as the head of the character object, etc. The target point and the number of target points can be set according to needs to track the trajectory of the character object.

[0084] The target tracking model can be a KCF model built based on the Kernel Correlation Filter (KCF) algorithm, and then the KCF model is used to track the human object. For example, in S1042, during the meeting period, the KCF model is used to track the trajectory based on the coordinates of the set target point. As an example, Figure 6 As shown, in order to improve the accuracy of determining the behavior event of the character object, before S1042, the method further includes the following steps:

[0085] S601, encoding the human body features of the human object in each video frame within the second target time period using a feature encoding algorithm;

[0086] S602, calculating the coding distance of the same human body feature in different video frames within the second target time period;

[0087] S603, comparing the coding distance with a preset threshold to determine whether the human objects in different video frames are the same human object;

[0088] S604: When the person objects in different video frames are the same person object, determine the motion trajectory information of the person object.

[0089] During the meeting, in the process of tracking the human objects, in order to accurately determine the motion trajectory information of the human objects, it is also necessary to determine the consistency of the identity of the same human object in different video clips. For example, in S601, the human body feature encoding algorithm is used to encode the human body features of the human objects in different video frames of each video clip, that is, vectorization processing is performed to obtain the vector information of the same human object in different video data, so as to encode the human body features of the human object, and use the encoding distance to perform similarity judgment on the vector information, thereby determining whether the human objects in different video data are the same human object, thereby laying the foundation for the information fusion of motion trajectory information and seat change information obtained based on multiple video data.

[0090] For example, in S602, the coding distance of the same human body features of the human objects in different video frames is calculated. The coding distance can be a cosine distance. The cosine similarity is used to characterize the similarity of the human objects in the two videos, as shown in the following formula (1). The cosine value of the angle between two vectors in the vector space is used as a measure of the difference:

[0091]

[0092] Among them, cosθ represents the cosine value of the angle between two vectors, A i and B i The feature vector representing the i-th feature of the person object in the two videos, n is a positive integer.

[0093] In S603, the preset threshold can be set to 0.4. When the angle θ between the two vectors approaches 0, the closer cosθ is to 1, the closer the two vectors are, the smaller the difference between the human objects is, and the higher the similarity between the human objects in the two videos is.

[0094] Therefore, in the process of tracking the person object, combined with the seat changes of the person object, the KCF model is used to track the motion trajectory of the person object, and the person objects in different video clips are authenticated. On the premise of determining that they are the same person object, the behavior event of the person object is determined based on the motion trajectory and seat change information of the person object. Since the target tracking model itself is a trained classifier with high confidence and strong response, it can track more accurate feature information based on different video frames, which can effectively improve the accuracy of behavioral event recognition.

[0095] In some embodiments, as Figure 7 As shown, if multiple behavioral events occur within the second target time period, the method further includes the following steps:

[0096] S701, determining the start time of every two adjacent behavior events;

[0097] S702: When the time difference between the start times of two adjacent behavior events is less than a preset value, merge the two adjacent behavior events into one behavior event.

[0098] As an example, the occurrence of a behavioral event is usually accompanied by a series of short events, especially the same event usually occurs simultaneously in the binocular camera, or occurs simultaneously in the binocular camera in part of the time period. Therefore, in the time dimension, the same person only generates one abnormal behavioral event at the same time, while different people are allowed to have multiple behavioral events in the same time period. Therefore, two adjacent behavioral events can be merged according to the start time and end time of the behavioral event. Specifically, if the time difference between the start time of each two adjacent behavioral events is less than the preset value, that is, the time interval between the occurrence of two adjacent behavioral events is less than the preset value, then the two adjacent events can be merged into one behavioral event.

[0099] For example, in the spatial dimension, the behavioral events generated by the human object can be identified and tracked. The identification can be an ID. If the event identification is detected to be the same, it is considered that the behavioral event is generated by the same human object. If there is an event identification that is not tracked, it can indicate that the human object has left the video detection range.

[0100] Therefore, when the time interval between adjacent behavior events is less than a preset value, the behavior events are merged, so as to obtain a complete behavior event sequence within the video clip time, and effectively avoid repeated output of behavior events.

[0101] In some embodiments, in S105, compared to a monocular camera scene, due to the different camera shooting angles in a binocular camera scene, the features of the human objects in the multiple video data obtained are different, so the motion trajectory information and seat change information obtained after tracking and identifying the human objects will also be different. Therefore, it is necessary to weightedly fuse the motion trajectory information and seat change information according to preset weights so as to more accurately determine the behavioral events of the human objects based on the fused information. The preset weights can be set to half for the motion trajectory information and half for the seat change information, and weighted fusion is performed based on weighted averaging.

[0102] As an example, behavioral events may include any of the following: entering, leaving, taking a seat, and leaving a seat. Seat change information may include any of the following: a person entering a meeting and wandering around, sitting down, changing seats, or being absent. "Entering" refers to a person appearing in the camera detection area, that is, entering the video clip, resulting in an increase in the number of people in the video. After entering the meeting room, the person is in a wandering state. If other target behaviors occur immediately after entering the meeting, the person may enter other states. "Leaving" refers to a person disappearing from the video screen, resulting in a decrease in the number of people in the screen. "Sitting" refers to a person sitting down at a seat identified during the sitting time period. If a person sits down at a seat identified during a non-sitting time period, it is deemed that no behavioral event has occurred. "Leaving" refers to a person leaving a seat identified during the sitting time period. If other behavioral events occur after leaving the seat, the person may enter other states.

[0103] Therefore, in the process of tracking and identifying human objects, even if cross-border behavior is involved, the motion trajectory of the human objects can be comprehensively identified with the help of the biological characteristics of the human objects and the seat coordinates in the conference scene, and the motion trajectory information and seat change information of the human objects in different video data can be integrated, thereby realizing weighted fusion of information cross-dimensionally and completing the identification of cross-shot behavioral events.

[0104] It should be noted that the application scenarios described in the above embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that as new application scenarios emerge, the technical solutions provided in the embodiments of the present application will also be applicable to similar technical problems.

[0105] Based on the same inventive concept, this application provides a behavior event recognition device, specifically combined with Figure 8 Provide detailed explanation.

[0106] Figure 8 A behavior event recognition device provided by an embodiment of the present application is shown. Figure 8 As shown, the apparatus 800 may include:

[0107] An acquisition module 801 is configured to acquire multiple video data of the same conference scene captured by multiple shooting devices at the same time from different angles;

[0108] The recognition module 802 is configured to perform object recognition on each video segment of the target time period in each video data to obtain the seat coordinates of the seat where the person object is located in the target area, where the seat coordinates include the initial seat and the target seat;

[0109] The first determining module 803 is configured to determine seat change information of a person object based on the seat coordinates of the initial seat and the seat coordinates of the target seat;

[0110] Tracking module 804, used to track the human object in the video clip and obtain the motion trajectory information of the human object;

[0111] The second determining module 805 is used to fuse the motion trajectory information and the seat change information to determine the behavior event of the character object.

[0112] Therefore, according to a behavioral event recognition device provided by an embodiment of the present application, multiple video data of the same conference scene shot from different angles at the same time are obtained, a video clip of the target time period is selected from each video data, and the video clip is subjected to object recognition to obtain the seat coordinates in the target area and the initial seat of the person object, and then the person object is tracked to obtain the motion trajectory information and seat change information of the person object, thereby fusing the motion trajectory information and seat change information to determine the behavioral event of the person object. In this process, the video data is shot from different angles and can more comprehensively reflect the visual features of the person object. When determining the behavioral event of the person object, the motion trajectory information and seat change information are fused, avoiding the recognition influence caused by occlusion or image superposition in the video, while solving the information fusion problem, and improving the accuracy of behavioral event recognition.

[0113] In some embodiments, the video data includes multiple video frames. To determine the coordinates of the seat located in the target area and the initial seat of the person object, the recognition module 802 may include the following submodules:

[0114] The recognition submodule is used to use the target detection model to recognize the image of the target area in the video frame, determine the seat coordinates of the seat where the person object is sitting in the target area, and the coordinates of the position where the person object and the seat are overlapping;

[0115] A first determining submodule is configured to determine an overlapping area between the person object and the seat where the person object is seated based on the seat coordinates of the seat where the person object is seated and the coordinates of the overlapping position between the person object and the seat where the person object is seated;

[0116] The second determining submodule is configured to determine the seat where the person object is seated based on an overlapping area between the person object and the seat where the person object is seated.

[0117] In some embodiments, if the identification result of the seat is not unique, the identification submodule may further include the following units:

[0118] an acquiring unit, configured to respectively acquire a first overlapping area between the person object and a first seat, and a second overlapping area between the person object and a second seat, where the first seat and the second seat are adjacent seats;

[0119] a comparing unit, configured to compare the first overlapping area, the second overlapping area, and a preset overlapping area threshold;

[0120] a first determining unit, configured to determine the first seat as a seat not occupied when the first overlapping area is greater than both the second overlapping area and a preset overlapping area threshold;

[0121] The second determining unit is configured to determine the second seat as a seat that has been taken if the second overlapping area is greater than both the first overlapping area and a preset overlapping area threshold.

[0122] In some embodiments, the tracking module 804 may include the following submodules:

[0123] A third determination submodule is configured to track the human object using the target tracking model during a second target period of the conference scene, and determine target point coordinates of the human object in different video frames during the second target period;

[0124] The fourth determining submodule is configured to obtain motion trajectory information of the human object based on the target point coordinates of the human object in each video frame within the second target time period.

[0125] In some embodiments, in order to determine the motion trajectory information of the human object, the fourth determining submodule may further include the following units:

[0126] an encoding unit, configured to encode the body features of the human object in each video frame within the second target time period using a feature encoding algorithm;

[0127] a calculation unit, configured to calculate coding distances of the same human features in different video frames within a second target time period;

[0128] A comparison unit, configured to compare the coded distance with a preset threshold value to determine whether the human objects in different video frames are the same human object;

[0129] The third determining unit is configured to determine the motion trajectory information and seat change information of the person object when the person objects in different video frames are the same person object.

[0130] In some embodiments, when there are multiple behavioral events, the tracking module 804 may further include the following submodules:

[0131] a fifth determining submodule, configured to determine the start time of each of two adjacent behavioral events;

[0132] The merging submodule is used to merge two adjacent behavior events into one behavior event when the time difference between the start times of the two adjacent behavior events is less than a preset value.

[0133] In some embodiments, the second determining module 805 may include the following submodules:

[0134] The weighted submodule is used to perform weighted fusion of the motion trajectory information and the seat change information according to preset weights to determine the behavioral events of the human object.

[0135] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0136] The electronic device 900 may include a processor 901 and a memory 902 storing computer program instructions.

[0137] Specifically, the processor 901 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0138] The memory 902 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 902 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 902 may include removable or non-removable (or fixed) media. Where appropriate, the memory 902 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 902 is a non-volatile solid-state memory.

[0139] The memory 902 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory 902 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the behavioral event recognition method according to the first aspect of the present application.

[0140] The processor 901 reads and executes computer program instructions stored in the memory 902 to implement any one of the behavior event recognition methods in the above embodiments.

[0141] In one example, the electronic device 900 may further include a communication interface 903 and a bus 904. Figure 9 As shown, the processor 901 , the memory 902 , and the communication interface 903 are connected via a bus 904 and communicate with each other.

[0142] The communication interface 903 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0143] Bus 904 comprises hardware, software or both, couples the parts of electronic device 900 to each other.For example, and not limitation, bus 904 may comprise accelerated graphics port (AGP) or other graphics buses, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 904 may comprise one or more buses.Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0144] The electronic device 900 can be combined with Figure 1 and Figure 8 Described is a method and apparatus for identifying behavioral events.

[0145] In addition, in conjunction with the behavioral event recognition method in the above embodiments, the present application embodiment may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the behavioral event recognition methods in the above embodiments is implemented.

[0146] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0147] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0148] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0149] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed via the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. This processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or the flowchart and the combination of the boxes in the block diagram and / or the flowchart can also be implemented by the dedicated hardware that performs the specified function or action, or can be implemented by the combination of dedicated hardware and computer instructions.

[0150] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A behavioral event recognition method, characterized in that: include: Acquire multiple video data of the same conference scene shot by multiple shooting devices at the same time from different angles; Performing object recognition on each video segment of the target time period in the video data to obtain the seat coordinates of the seat where the person object is located in the target area, wherein the seat where the person object is located includes the initial seat where the person object is located and the target seat where the person object is located; Determining seat change information of the person object based on the seat coordinates of the initial seat and the seat coordinates of the target seat; Tracking the person object in the video clip to obtain motion trajectory information of the person object; The motion trajectory information and the seat change information are integrated to determine the behavior event of the character object.

2. The method according to claim 1, characterized in that The video clip includes a plurality of video frames, and the object recognition is performed on each video clip of the target time period in the video data to obtain the seat coordinates of the seat where the person object is located in the target area, including: Using a target detection model to identify the image of the target area in the video frame, and determining the seat coordinates of the seat where the person object is seated in the target area and the coordinates of the overlapping position between the person object and the seat; Determining an overlapping area between the person object and the seat based on the seat coordinates of the seat where the person object is seated and the coordinates of the overlapping position between the person object and the seat where the person object is seated; The seat where the person object is seated is determined based on the overlapping area between the person object and the seat where the person object is seated.

3. The method according to claim 2, characterized in that If the identification result of the seat is not unique, the method further includes: respectively obtaining a first overlapping area between the person object and a first seat, and a second overlapping area between the person object and a second seat, where the first seat and the second seat are adjacent seats; comparing the first overlapping area, the second overlapping area, and a preset overlapping area threshold; When the first overlapping area is greater than both the second overlapping area and the preset overlapping area threshold, determining the first seat as the unoccupied seat; When the second overlapping area is greater than both the first overlapping area and the preset overlapping area threshold, the second seat is determined as the seat where the passenger is seated.

4. The method according to claim 1, wherein Tracking the person object in the video clip to obtain motion trajectory information of the person object includes: During a second target period of the conference scene, the person object is tracked using a target tracking model to determine target point coordinates of the person object in different video frames during the second target period; Based on the target point coordinates of the human object in each of the video frames within the second target time period, the motion trajectory information of the human object is obtained.

5. The method according to claim 4, characterized in that Before obtaining the motion trajectory information of the human object based on the target point coordinates of the human object in each of the video frames within the second target time period, the method further includes: Encoding the human body features of the human object in each of the video frames within the second target time period using a feature encoding algorithm; Calculating the coding distance of the same human body feature in different video frames within the second target time period; Comparing the encoding distance with a preset threshold to determine whether the human objects in different video frames are the same human object; In a case where the person objects in different video frames are the same person object, the motion trajectory information of the person object is determined.

6. The method according to claim 1, characterized in that When there are multiple behavioral events, the method further includes: Determining the start time of each two adjacent behavioral events; When the time difference between the start times of two adjacent behavior events is less than a preset value, the two adjacent behavior events are merged into one behavior event.

7. The method according to claim 1, characterized in that The fusing of the motion trajectory information and the seat change information to determine the behavior event of the character object includes: The motion trajectory information and the seat change information are weightedly fused according to preset weights to determine the behavior event of the character object.

8. A behavioral event recognition device, characterized in that: include: An acquisition module is used to acquire multiple video data of the same conference scene shot by multiple shooting devices at the same time from different angles; an identification module, configured to perform object identification on each video segment of a target time period in the video data, and obtain the seat coordinates of a seat where a person object is located in a target area, wherein the seat where a person is located includes an initial seat where a person is located and a target seat where a person is located; A first determining module is configured to determine seat change information of the person object based on the seat coordinates of the initial seat and the seat coordinates of the target seat; A tracking module, configured to track the person object in the video clip to obtain motion trajectory information of the person object; The second determining module is configured to fuse the motion trajectory information and the seat change information to determine the behavior event of the character object.

9. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the behavior event recognition method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the behavior event recognition method according to any one of claims 1 to 7.

Images