Computer vision based behavior recognition method

Abstract

The application relates to the technical field of computer vision, in particular to a behavior recognition method based on computer vision, which comprises the following steps: adopting a behavior recognition model to recognize a plurality of action categories in any target monitoring video, and taking each action category as an action behavior label; based on a preset background model, acquiring a static image set corresponding to each action duration period, and further obtaining an action region label corresponding to each action duration period; acquiring an initial scene description text corresponding to each action duration period and obtaining a plurality of target scene description texts through fusion processing; inputting each target scene description text into a preset classification model to obtain an event classification result, so as to judge whether a preset behavior event occurs in the target monitoring video; the application can quickly and accurately recognize whether the preset behavior event occurs, and improves the judgment efficiency and accuracy of the preset behavior event.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a behavior recognition method based on computer vision. BACKGROUND

[0002] At present, prohibited behavior events are set in many scene spaces, such as prohibition of open fire in warehouses, prohibition of smoking in office areas, etc. The detection of behavior events in the scene mainly depends on sensor monitoring, video analysis and text rule matching. For simple scenes and simple events that are easy to identify, behavior events can be accurately detected by sensor monitoring, but are easily disturbed by the environment. Text rule matching can judge events through keyword matching, but has poor generalization ability and cannot understand the context semantics, so video analysis method is usually adopted. The video analysis based on the computer vision monitoring system can recognize actions, but for complex scenes and events that are not easy to distinguish, the misjudgment rate is high due to the influence of complex environment and various actions. If the monitoring video is checked one by one by artificial, the accuracy can be improved, but the recognition efficiency is greatly reduced.

[0003] Therefore, there is an urgent need for a behavior recognition method which can quickly and accurately recognize behavior events. SUMMARY

[0004] In view of the above technical problems, the present application provides a behavior recognition method based on computer vision, which can quickly recognize whether a preset behavior event occurs on the basis of recognizing action categories by computer vision technology and a series of processing of monitoring video, thereby improving the determination efficiency and accuracy of the preset behavior event.

[0005] The present application provides a behavior recognition method based on computer vision, comprising the following steps:

[0006] S100, acquiring a plurality of target monitoring videos corresponding to a target region in a historical time period; different target monitoring videos correspond to different preset time intervals.

[0007] S200, for any target monitoring video, using a behavior recognition model to recognize a plurality of action categories in the target monitoring video, and taking each action category as an action behavior label; each action behavior label corresponds to an action duration period.

[0008] S300, based on a preset background model of the target region, acquiring a static image set corresponding to each action duration period, and obtaining an action region label corresponding to each action duration period according to the static image set corresponding to each action duration period.

[0009] S400, according to the action behavior label and the action region label corresponding to each action duration period, obtaining initial scene description texts corresponding to each action duration period, and fusing the initial scene description texts to obtain target scene description texts.

[0010] S500, inputting each target scene description text into a preset classification model to obtain an event classification result, and determining whether the preset behavior event occurs in the target monitoring video according to the event classification result; the preset classification model is obtained by optimizing a preset pre-training language model.

[0011] The application has at least the following beneficial effects:

[0012] The application provides a behavior recognition method based on computer vision, which first obtains a plurality of target monitoring videos corresponding to a target region in a historical time period, identifies a plurality of action categories in any target monitoring video by using a behavior recognition model, and takes each action category as an action behavior label, so as to realize preliminary understanding of the monitoring video picture and facilitate obtaining the occurrence of an event according to the dynamic situation of the monitoring video picture; then, based on a preset background model, a static image set corresponding to each action duration period is obtained, and an action region label corresponding to each action duration period is further obtained according to the static image set, so as to provide a further judgment basis for determining whether the preset behavior event occurs; then, initial scene description texts corresponding to each action duration period are obtained and fused to obtain a plurality of target scene description texts, so as to reduce the subsequent data processing amount and improve the scene description accuracy of the target scene description texts, and facilitate obtaining a more reliable event classification result; finally, each target scene description text is input into a preset classification model to obtain an event classification result, so as to determine whether the preset behavior event occurs in the target monitoring video, so that the historical time period corresponding to the target monitoring video is quickly determined whether the preset behavior event occurs, and the determination efficiency and accuracy of the preset behavior event are improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0014] Figure 1 The flowchart of the behavior recognition method based on computer vision provided by the embodiments of the application. DETAILED DESCRIPTION

[0015] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0016] The present application provides a computer vision-based behavior recognition method, as shown in Figure 1 The method comprises the following steps:

[0017] S100, acquiring a plurality of target monitoring videos corresponding to a target area in a historical time period; different target monitoring videos correspond to different preset time intervals. For example, the historical time period is any day of last month, and the target monitoring video is a monitoring video corresponding to any preset time interval in a day.

[0018] In one implementation scenario, the target area can be a restaurant kitchen, different preset time intervals can be business hours such as 8-10 o'clock and 12-14 o'clock, and the target area includes a plurality of small areas, such as a cooking area and a food preparation area, etc., the purpose is to determine whether the restaurant kitchen has a second meal serving behavior in a complex kitchen environment.

[0019] S200, for any target monitoring video, a behavior recognition model is used to identify a plurality of action categories in the target monitoring video, and each action category is taken as an action behavior label; wherein the behavior recognition model is a computer vision technology, and those skilled in the art know the specific implementation of identifying action categories by using the behavior recognition model, for example, identifying action categories such as walking, falling, waving hands, etc.

[0020] Specifically, each action behavior label corresponds to an action duration period; it can be understood that: the action behavior label corresponds to a start time and an end time of the action, and the action duration period is between the start time and the end time.

[0021] Through the above, by identifying a plurality of action categories and action duration periods in the monitoring video, it can be known that what action occurs in what period, preliminary understanding of the monitoring video picture is achieved, which is beneficial to know the occurrence of an event according to the dynamic situation of the monitoring video picture, for example, whether a second meal serving behavior occurs.

[0022] S300, based on a preset background model of the target area constructed in advance, a static image set corresponding to each action duration period is acquired, and an action area label corresponding to each action duration period is obtained according to the static image set corresponding to each action duration period; it can be understood that: the area label corresponding to the range where the person acts is acquired, for example, the cooking area.

[0023] Specifically, the static image set corresponding to each action duration period is obtained through the following steps:

[0024] S301, obtaining a plurality of frame initial monitoring images corresponding to each action duration period from the target monitoring video; it can be understood that based on the action duration, the dynamic video corresponding to the action duration period and each frame image in the dynamic video are obtained from the target monitoring video.

[0025] S302, determining the dynamic area pixel points corresponding to each frame initial monitoring image according to the dynamic range of the target monitoring video corresponding to each action duration period.

[0026] S303, separating the dynamic area pixel points corresponding to each frame initial monitoring image from the corresponding initial monitoring image, and performing pixel completion on the separated pixel point position in the initial monitoring image based on the preset background model to obtain the target monitoring image corresponding to the initial monitoring image, so as to obtain the static image set corresponding to each action duration period; it can be understood that the plurality of frame target monitoring images corresponding to the action duration period are combined into the static image set corresponding to the action duration period itself.

[0027] It should be noted that when the static object in the initial monitoring image changes relative to the background image corresponding to the preset background model, the target monitoring image also changes relative to the background image corresponding to the preset background model.

[0028] Further, the action area label corresponding to each action duration period is obtained through the following steps:

[0029] S310, for any frame static image corresponding to any action duration period, obtaining the pixel completion area in the static image.

[0030] S320, comparing the pixel completion area in the static image with the background image corresponding to the preset background model, obtaining the boundary static object of the pixel completion area in the static image according to the comparison result, and performing region expansion on the edge of the pixel completion area in the static image based on the boundary static object of the pixel completion area, to obtain the to-be-recognized object area corresponding to the static image. For example, it is known that the pixel completion area is the part of the background area blocked by the person who acts, and the purpose is to only obtain the area where the person acts, but there are incomplete objects at the edge critical point of the pixel completion area, such as only half of the dinner plate or wok, at this time, the background image of the dinner plate is needed to be used to expand the pixel completion area, so that the to-be-recognized object area displays a complete dinner plate or wok.

[0031] S330, same static goods as the image region corresponding to the preset background model are identified from the to-be-identified goods region corresponding to each frame of static image, and based on a preset correspondence relationship table of static goods and action region labels, an action region label corresponding to the static goods identified from each frame of static image is determined.

[0032] Specifically, the image region corresponding to the preset background model refers to a region in the background image corresponding to the preset background model and corresponding to the to-be-identified goods region.

[0033] S340, according to the several frames of static images corresponding to each action duration period and the action region label corresponding to the static goods identified from each frame of static image, several action region labels corresponding to each action duration period are obtained.

[0034] In the above, under the premise of needing to identify whether a preset behavior event occurs in the action duration period, the original background image corresponding to the pixel point region where the action occurs is also obtained through a series of image processing, that is, the to-be-identified goods region, a region consistent with the to-be-identified goods region is divided from the background image corresponding to the preset background model, and the same static goods are found through goods comparison, which are used to determine the action region label and provide further judgment basis for judging whether the preset behavior event occurs.

[0035] S400, according to the action behavior label and the action region label respectively corresponding to each action duration period, an initial scene description text corresponding to each action duration period is obtained, and a plurality of initial scene description texts are fused to obtain a plurality of target scene description texts.

[0036] Specifically, the initial scene description text corresponding to the action duration period includes a time point corresponding to the action duration period, an action region label, a person, and an action detail corresponding to the action behavior label; it can be understood that: the initial scene description text refers to a semantic description of the action occurring in the action duration period and the scene. The action detail corresponding to the action behavior label refers to the specific details of the action, for example, walking from the first region to the second region, flipping the spoon for 5 times, etc.

[0037] Further, the fusion processing of the plurality of initial scene description texts to obtain the plurality of target scene description texts includes the following steps:

[0038] S401, the initial scene description text corresponding to each action duration period is vector converted, and the text similarity between the initial scene description texts corresponding to each adjacent two action duration periods is calculated based on the vector conversion result.

[0039] S402, when there is a text similarity greater than a preset similarity threshold and there is no continuous text similarity greater than the preset similarity threshold, performing semantic fusion on two initial scene description texts corresponding to the text similarity greater than the preset similarity threshold to obtain a target scene description text; it can be understood that the target scene description text is consistent with the elements corresponding to the initial scene description text, but the content corresponding to the elements has changed, for example, the time point corresponding to the action duration period and the start time and end time of the action have all changed.

[0040] S403, when there is continuous text similarity greater than the preset similarity threshold, performing semantic fusion on several initial scene description texts corresponding to the continuous text similarity greater than the preset similarity threshold to obtain a target scene description text; it can be understood that the continuous similar multiple actions are fused into one action category, for example, different angle stirring actions during cooking are regarded as one action category, and multiple continuous serving actions are regarded as one action category.

[0041] S404, filtering out initial scene description texts that have not been fused from the several initial scene description texts, and taking the filtered initial scene description texts as target scene description texts.

[0042] The above, by calculating the similarity of adjacent initial scene description texts, continuous and similar actions can be fused into one action category, reducing the number of scene description texts, and further reducing the amount of subsequent data processing, and also making the action events described by the obtained target scene description text more comprehensive and improving the scene description accuracy of the target scene description text, and further improving the accuracy and reliability of the event classification result.

[0043] S500, inputting each target scene description text into a preset classification model to obtain an event classification result, so as to determine whether a preset behavior event occurs in the target monitoring video according to the event classification result; the preset classification model is obtained by optimizing a preset pre-training language model; it can be understood that a full connection layer is configured in the output layer of the pre-training language model, and the pre-training language model is optimized into a classification model.

[0044] In one specific embodiment, the step S500 includes the following steps:

[0045] S501, input the target scene description text into a preset pre-trained language model, and obtain a context vector corresponding to the target scene description text through the pre-trained language model. It should be noted that the pre-trained language model is a model trained according to a labeled data set. The labeled data set includes a large number of scene description text samples and a text label corresponding to each scene description text sample. The text label is 0 or 1. 0 represents that a preset behavior event does not occur, and 1 represents that the preset behavior event occurs. Those skilled in the art know the specific training process of the pre-trained language model, which is not repeated here.

[0046] S502, processing the context vector corresponding to the target scene description text by using the pre-trained language model, and outputting a scene information vector corresponding to the target scene description text. In this embodiment, the pre-trained language model uses a BERT model, and the scene information vector is output through the [CLS] label of the BERT model.

[0047] S503, inputting the scene information vector corresponding to the target scene description text into a full connection layer to output an event classification result through the full connection layer; it can be understood that the scene information vector is mapped to two dimensions through the full connection layer, which is equivalent to a classification layer, to obtain the event classification result.

[0048] Specifically, the event classification result refers to occurrence of a preset behavior event or non-occurrence of the preset behavior event. For example, when judging whether the behavior of secondarily cooking remaining food occurs, when the action of pouring the remaining food into a pot and heating it again, or the action of adding food to a food tray in the back kitchen, it is considered that the preset behavior event occurs; those skilled in the art can set the judgment rule according to the actual demand, which is not repeated here.

[0049] In the above, the classification layer is added in the pre-trained language model to realize fine tuning of the pre-trained language model, the pre-trained language model is converted into a binary classification model, the scene information vector corresponding to the target scene description text is processed after semantic understanding of the target scene description text, and then whether the event corresponding to the target scene description text is the preset behavior event can be accurately judged, so that whether the preset behavior event occurs in the historical time period corresponding to the target monitoring video can be accurately and quickly detected, without manual checking of the video, and therefore the determination efficiency is improved.

[0050] Further, the method further includes the following steps:

[0051] S1, based on the obtained several video samples corresponding to the preset behavior event, each video sample is disassembled into several discrete action units. In specific implementation, the video sample can be disassembled according to the preset key action corresponding node.

[0052] S2, for any two adjacent action units, a preset correlation coefficient between the two action units is taken as a dependent variable, and time interval indicators, position distance indicators and action similarity indicators corresponding to the two action units are taken as independent variables, and are substituted into a preset correlation coefficient evaluation model to be optimized to obtain an intermediate correlation coefficient evaluation model; wherein, initial values of influence factors of each independent variable in the correlation coefficient evaluation model to be optimized are unknown. In specific implementation, the preset correlation coefficient between the two action units is set by an expert according to the correlation of the actions.

[0053] It should be noted that when the correlation coefficient evaluation model to be optimized is y=ax1+bx2+cx3, y represents the preset correlation coefficient, x1, x2 and x3 are respectively the time interval indicator, the position distance indicator and the action similarity indicator corresponding to the two action units, a, b and c are respectively the influence factors corresponding to the time interval indicator, the position distance indicator and the action similarity indicator, and the obtained intermediate correlation coefficient evaluation model is a function after the independent variables are substituted. Among them, the position distance indicator can be obtained according to the center distance of the two regions where the two actions are located, and the action similarity can be determined according to the text vector similarity corresponding to the action behavior label.

[0054] S3, based on the plurality of intermediate correlation coefficient evaluation models corresponding to the plurality of video samples, the influence factors of each indicator in the correlation coefficient evaluation model to be optimized are obtained by an optimization algorithm to obtain a target correlation coefficient evaluation model. In specific implementation, the optimization algorithm can adopt a genetic algorithm or a grid optimization algorithm, and the process of optimization according to the genetic algorithm or the grid optimization algorithm is a prior art, which will not be described here.

[0055] S4, the time interval, the position distance and the action similarity between each two adjacent action behavior labels in the target scene description text are respectively input into the target correlation coefficient evaluation model to obtain the correlation coefficient evaluation value corresponding to each two adjacent actions.

[0056] S5, the target scene description text and the correlation coefficient evaluation value corresponding to the target scene description text are input into a preset classification model to obtain an event classification result. For example, when the target scene description text is obtained after fusion processing of the three initial scene description texts, the target scene description text corresponds to three action units, that is, two correlation coefficient evaluation values.

[0057] It should be noted that in the above embodiment, when the preset classification model is trained, the correlation coefficient evaluation value between adjacent actions should also be introduced into the training data set for training.

[0058] The above, since the target scene description text corresponds to at least two actions, considering that there are similar actions but the events may be completely inconsistent, and the video duration corresponding to different texts is inconsistent, all of which will interfere with the judgment result, the application not only inputs the scene description text, but also inputs the correlation coefficient between different actions, considers the multi-dimensional differences between different actions, so that the final classification result is more reliable.

[0059] Further, the method further comprises the following steps:

[0060] S10, if it is determined that the preset behavior event occurs in the target monitoring video, the action duration period corresponding to the occurrence of the preset behavior event is obtained.

[0061] S20, according to the action duration period corresponding to the occurrence of the preset behavior event, the action duration video is separated from the target monitoring video and sent to the user for review; it can be understood that the action duration video refers to the video consistent with the time point of the action duration period corresponding to the occurrence of the preset behavior event.

[0062] The above, although a series of video processing and text processing greatly improve the preset behavior event determination efficiency and accuracy, but there are still small errors, at this time the corresponding video segment is sent to the user, so that the user reviews whether the preset behavior event occurs, and only the video corresponding to the time period when the preset behavior event is determined to occur is sent to the user, so that the determination result is more accurate through manual review under the condition of only increasing a small amount of workload.

[0063] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A computer vision-based behavior recognition method, characterized by, The method comprises the following steps: S100, acquiring a plurality of target monitoring videos corresponding to a target region in a historical time period; different target monitoring videos correspond to different preset time intervals; S200, for any target monitoring video, using a behavior recognition model to recognize a plurality of action categories in the target monitoring video, and taking each action category as an action behavior label; each action behavior label corresponds to an action duration period; S300, based on a preset background model of the target region constructed in advance, acquiring a static image set corresponding to each action duration period, and obtaining an action region label corresponding to each action duration period according to the static image set corresponding to each action duration period; wherein the action region label corresponding to each action duration period is obtained by the following steps: S310, for any frame of static image corresponding to any action duration period, acquiring a pixel completion region in the static image; the pixel completion region in the static image is a region obtained by performing pixel completion on the position of the dynamic region pixel points separated in any initial monitoring image corresponding to any action duration period based on the preset background model; S320, comparing the pixel completion region in the static image with a background image corresponding to the preset background model, obtaining a boundary static article of the pixel completion region in the static image according to the comparison result, and performing region expansion on the edge of the pixel completion region in the static image based on the boundary static article, to obtain a to-be-recognized article region corresponding to the static image; S330, respectively recognizing static articles identical to an image region corresponding to the preset background model from the to-be-recognized article region corresponding to each frame of static image, and determining an action region label corresponding to the static article recognized from each frame of static image based on a preset correspondence relationship table of static articles and action region labels; S340, obtaining a plurality of action region labels corresponding to each action duration period according to a plurality of frames of static images corresponding to each action duration period and the action region label corresponding to the static article recognized from each frame of static image; S400, acquiring an initial scene description text corresponding to each action duration period according to the action behavior label and the action region label corresponding to each action duration period respectively, and performing fusion processing on a plurality of initial scene description texts to obtain a plurality of target scene description texts; S500, inputting each target scene description text into a preset classification model to obtain an event classification result, so as to determine whether a preset behavior event occurs in the target monitoring video according to the event classification result; the preset classification model is obtained by optimizing a preset pre-training language model.

2. The computer vision based behavior recognition method of claim 1, wherein, In the S300 step, the static image set corresponding to each action duration period is acquired by the following steps: S301, acquiring a plurality of frames of initial monitoring images corresponding to each action duration period from the target monitoring video; S302, determining a dynamic region pixel point corresponding to each frame of initial monitoring image according to a dynamic range in the target monitoring video corresponding to each action duration period; S303, separate the pixel points of the dynamic region corresponding to each initial monitoring image from the initial monitoring image, and perform pixel completion on the separated pixel points in the initial monitoring image based on a preset background model to obtain a target monitoring image corresponding to the initial monitoring image, so as to obtain a static image set corresponding to each action duration period.

3. The computer vision based behavior recognition method of claim 1, wherein, The initial scene description text corresponding to the action duration period includes an action detail corresponding to a time point, an action region label, a person, and an action behavior label of the action duration period.

4. The computer vision based behavior recognition method of claim 1, wherein, In the S400 step, the fusion processing of the plurality of initial scene description texts is performed to obtain a plurality of target scene description texts, including the following steps: S401, vector conversion is performed on the initial scene description text corresponding to each action duration period, and the text similarity between the initial scene description texts corresponding to each two adjacent action duration periods is calculated based on the vector conversion result; S402, when there is a text similarity greater than a preset similarity threshold and there is no continuous text similarity greater than the preset similarity threshold, the two initial scene description texts corresponding to the text similarity greater than the preset similarity threshold are semantically fused to obtain a target scene description text; S403, when there is continuous text similarity greater than the preset similarity threshold, the plurality of initial scene description texts corresponding to the continuous text similarity greater than the preset similarity threshold are semantically fused to obtain a target scene description text; S404, the initial scene description text that has not been fused is screened out from the plurality of initial scene description texts, and the screened initial scene description text is taken as a target scene description text.

5. The computer vision based behavior recognition method of claim 1, wherein, The S500 step includes the following steps: S501, inputting the target scene description text into a preset pre-trained language model to obtain a context vector corresponding to the target scene description text through the pre-trained language model; S502, processing the context vector corresponding to the target scene description text by using the pre-trained language model to output a scene information vector corresponding to the target scene description text; S503, inputting the scene information vector corresponding to the target scene description text into a full connection layer to output an event classification result through the full connection layer; The event classification result refers to occurrence of a preset behavior event or non-occurrence of the preset behavior event.

6. The computer vision based behavior recognition method of claim 1, wherein, The method further includes the following steps: S1, based on the plurality of video samples corresponding to the preset behavior event, each video sample is disassembled into a plurality of discrete action units; S2, for any two adjacent action units, a preset correlation coefficient between the two action units is taken as a dependent variable, and a time interval index, a position distance index, and an action similarity index corresponding to the two action units are taken as independent variables, which are substituted into a preset to-be-optimized correlation coefficient evaluation model to obtain an intermediate correlation coefficient evaluation model; wherein the initial value of the influence factor of each independent variable in the to-be-optimized correlation coefficient evaluation model is unknown; S3, based on the plurality of intermediate correlation coefficient evaluation models corresponding to the plurality of video samples, the influence factor of each index in the to-be-optimized correlation coefficient evaluation model is obtained through an optimization algorithm to obtain a target correlation coefficient evaluation model; S4, input the time interval, the position distance and the action similarity between each two adjacent action behavior tags in the target scene description text into the target correlation coefficient evaluation model respectively, to obtain the correlation coefficient evaluation value corresponding to each two adjacent action pairs; S5, input the target scene description text and the correlation coefficient evaluation value corresponding to the target scene description text into the preset classification model, to obtain the event classification result.

7. The computer vision based behavior recognition method of claim 1, wherein, The method further comprises the following steps: S10, if it is determined that the preset behavior event occurs in the target monitoring video, the action duration period corresponding to the occurrence of the preset behavior event is obtained; S20, according to the action duration period corresponding to the occurrence of the preset behavior event, the action duration video is separated from the target monitoring video and sent to the user for review.

Images