Image-based behavior recognition method, device, equipment and storage medium

By performing object detection and image quality scoring on video frame images and screening high-quality images for behavior recognition, the problem of long-term detection in the prior art is solved, and the efficiency and accuracy of behavior recognition are improved.

CN111797733BActive Publication Date: 2024-12-03ZHEJIANG DAHUA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010575337.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-22
Publication Date
2024-12-03
Estimated Expiration
2040-06-22

AI Technical Summary

Technical Problem

In the prior art, detecting video streams or picture sequences during behavior recognition is longer, which affects efficiency.

Method used

By target detection of each frame of images in the video to be detected, the image characteristics of the target object are obtained, the image quality is determined, and images that meet the preset quality requirements are selected for behavior recognition. The weight weighted sum score is used to filter low quality Images to improve recognition efficiency.

Benefits of technology

It realizes the complex way of identifying images that need to be recognized, and filters out images that do not need to be recognized, thereby improving recognition efficiency, avoiding the adverse effects of low-quality images on the results, and improving accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111797733B_ABST
    Figure CN111797733B_ABST
Patent Text Reader

Abstract

The present disclosure provides an image-based behavior recognition method, device, equipment and storage medium, the method comprising: performing target detection on each frame image in a video to be detected to obtain image features of the target object; determining the image quality of each frame image according to the image features of the target object; screening out images whose image quality meets preset quality requirements, and performing behavior recognition on the target object. The entire process does not require that each frame image in the video be used for behavior recognition, only images that meet the preset quality requirements need to be subjected to behavior recognition. Therefore, it is possible to implement complex behavior recognition methods for images that require behavior recognition, while filtering out some images that do not need to be recognized, thereby improving recognition efficiency. In addition, by performing behavior recognition on high-quality images, it is possible to avoid the problem that low-quality images are prone to behavior recognition errors that adversely affect the overall behavior recognition results.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] With the development of computer vision, artificial intelligence, machine learning and other technologies, the concept of smart education has emerged, and at the same time, there is a demand for target behavior recognition in classroom teaching videos. At present, behavior analysis and recognition technology has been widely used in video surveillance, human-computer interaction, virtual reality and other fields, and has gradually become a research hotspot in the field of computer vision. Behavior recognition technology uses a series of algorithms to enable computers to identify and analyze the behavior of targets in video streams or image sequences, and finally obtain recognition results that meet expectations.

[0003] In the prior art, when performing behavior recognition, the input video stream or picture sequence is detected, but the detection takes a long time. Therefore, a new method is urgently needed to solve the above problem. Summary of the invention

[0004] The present disclosure provides an image-based behavior recognition method, device, equipment and storage medium, which are used to solve the problem of long detection time in the prior art.

[0005] In a first aspect, the present disclosure provides an image-based behavior recognition method, the method comprising:

[0006] Perform target detection on each frame image in the video to be detected to obtain the image features of the target object;

[0007] Determining the image quality of each frame of image according to the image features of the target object;

[0008] The images whose image quality meets the preset quality requirement are screened out, and behavior recognition for the target object is performed.

[0009] In one embodiment, the image features include the deflection angle of the target object and the image clarity of the target object; wherein,

[0010] The deflection angle of the target object is used to identify the angle between the facial orientation of the target object and the lens plane of the camera;

[0011] Determining the image quality of each frame of image according to the image feature of the target object includes:

[0012] According to a preset image quality scoring rule, the image quality score of each frame of image under the deflection angle of the target object and the image clarity of the target object is determined.

[0013] In one embodiment, determining the image quality score of each frame of image at the deflection angle of the target object and the image clarity of the target object according to a preset image quality scoring rule includes:

[0014] Performing a weighted sum according to the weight of the deflection angle of the target object and the weight of the image clarity of the target object to obtain a score of the image quality;

[0015] The weight of the deflection angle of the target object and the weight of the image definition of the target object are obtained by training a first neural network, and the method further includes:

[0016] The first neural network is trained by:

[0017] Acquire a first image feature training sample, wherein the first image feature training sample includes a deflection angle of a target object of each frame image, an image clarity of the target object, an initial weight of the deflection angle of the target object, an initial weight of the image clarity of the target object, and a preset score of each frame image marked in advance;

[0018] Inputting the first image feature training sample into the first neural network, scoring each frame of the image according to the deflection angle of the target object of each frame of the image and the image clarity of the target object, and obtaining a score of the image quality of each frame of the image predicted by the first neural network;

[0019] Comparing the obtained image quality score of each frame of image with the preset score of each frame of image to obtain a loss value;

[0020] The first neural network is trained according to the loss value to obtain a weight of the deflection angle of the target object and a weight of the image clarity.

[0021] In one embodiment, before determining the image quality of each frame of image according to the image feature of the target object, the method further includes:

[0022] Tracking the target object according to the video to be detected to obtain a motion trajectory of the target object;

[0023] If the target object is determined to be a moving target according to the motion trajectory, a step of determining the image quality of each frame image according to the image features of the target object is performed.

[0024] In one embodiment, the image features further include confidence;

[0025] Before determining the image quality of each frame of image according to the image feature of the target object, the method further includes:

[0026] For any frame image, if the confidence of the deflection angle of the target object of the image and the confidence of the image clarity of the target object do not meet specified conditions, the image is filtered; the specified conditions include that the confidence of the deflection angle of the target object is greater than a first preset threshold and the confidence of the image clarity of the target object is greater than a second preset threshold.

[0027] In one embodiment, the image features further include contour information;

[0028] The step of screening out the images whose image quality meets the preset quality requirement and performing behavior recognition on the target object includes:

[0029] Acquire the contour information of the target object in the screened image; input the screened image into a second neural network for behavior recognition to obtain behavior features; and,

[0030] Inputting the screened image into a third neural network for face recognition to obtain facial attribute features;

[0031] The behavior feature matrix obtained by combining the facial attribute features with the behavior features is matched with a pre-labeled behavior feature matrix to obtain a final behavior recognition result.

[0032] In a second aspect, the present disclosure provides an image-based behavior recognition device, the device comprising:

[0033] The target detection module is used to perform target detection on each frame image in the video to be detected to obtain the image features of the target object;

[0034] An image quality determination module, used to determine the image quality of each frame of image according to the image features of the target object;

[0035] The behavior recognition module is used to screen out the images whose image quality meets the preset quality requirements and perform behavior recognition on the target object.

[0036] In one embodiment, the image features include the deflection angle of the target object and the image clarity of the target object; wherein,

[0037] The deflection angle of the target object is used to identify the angle between the facial orientation of the target object and the lens plane of the camera;

[0038] The image quality determination module is further used for:

[0039] According to a preset image quality scoring rule, the image quality score of each frame of image under the deflection angle of the target object and the image clarity of the target object is determined.

[0040] In one embodiment, the image quality determination module is further used to:

[0041] Performing a weighted sum according to the weight of the deflection angle of the target object and the weight of the image clarity of the target object to obtain a score of the image quality;

[0042] The weight of the deflection angle of the target object and the weight of the image definition of the target object are obtained by training a first neural network, and the device further includes:

[0043] A training sample acquisition module is used to train the first neural network by the following method: acquiring a first image feature training sample, wherein the first image feature training sample includes a deflection angle of a target object of each frame image, an image clarity of the target object, an initial weight of the deflection angle of the target object, an initial weight of the image clarity of the target object, and a preset score of each frame image marked in advance;

[0044] a scoring module, configured to input the first image feature training sample into the first neural network, score each frame of the image according to the deflection angle of the target object of each frame of the image and the image clarity of the target object, and obtain a score of the image quality of each frame of the image predicted by the first neural network;

[0045] A comparison module, used for comparing the obtained image quality score of each frame of image with the preset score of each frame of image to obtain a loss value;

[0046] A weight acquisition module is used to train the first neural network according to the loss value to obtain the weight of the deflection angle of the target object and the weight of the image clarity.

[0047] In one embodiment, the apparatus further comprises:

[0048] A tracking module, used for tracking the target object according to the video to be detected to obtain a motion trajectory of the target object before determining the image quality of each frame of the image according to the image features of the target object;

[0049] The moving target determination module is used to execute the step of determining the image quality of each frame image according to the image features of the target object if the target object is determined to be a moving target according to the motion trajectory.

[0050] In one embodiment, the image feature further includes a confidence level; and the device further includes:

[0051] A filtering module is used to filter the image before determining the image quality of each frame of the image based on the image features of the target object. For any frame of the image, if the confidence of the deflection angle of the target object and the confidence of the image clarity of the target object do not meet specified conditions, the specified conditions include that the confidence of the deflection angle of the target object is greater than a first preset threshold and the confidence of the image clarity of the target object is greater than a second preset threshold.

[0052] In one embodiment, the image features further include contour information;

[0053] The behavior recognition module is further used for:

[0054] Acquire the contour information of the target object in the screened image; input the screened image into a second neural network for behavior recognition to obtain behavior features; and,

[0055] Inputting the screened image into a third neural network for face recognition to obtain facial attribute features;

[0056] The behavior feature matrix obtained by combining the facial attribute features with the behavior features is matched with a pre-labeled behavior feature matrix to obtain a final behavior recognition result.

[0057] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0058] According to a fifth aspect provided by an embodiment of the present disclosure, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.

[0059] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0060] The present disclosure provides an image-based behavior recognition method, device, equipment and storage medium. The method includes: performing target detection on each frame image in the video to be detected to obtain the image features of the target object; determining the image quality of each frame image according to the image features of the target object; screening out the images whose image quality meets the preset quality requirements, and performing behavior recognition on the target object. The entire process does not require that each frame image in the video be used for behavior recognition, only the images that meet the preset quality requirements need to be subjected to behavior recognition. Therefore, it is possible to implement complex behavior recognition methods for images that require behavior recognition, while filtering out some images that do not need to be recognized, thereby improving recognition efficiency. In addition, by performing behavior recognition on high-quality images, it is possible to avoid the problem that low-quality images are prone to behavior recognition errors that adversely affect the overall behavior recognition results.

[0061] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0063] Figure 1 A schematic diagram of an applicable scenario in an embodiment of the present disclosure;

[0064] Figure 2 This is one of the example diagrams of image-based behavior recognition according to one embodiment of the present disclosure;

[0065] Figure 3 This is a second example diagram of image-based behavior recognition according to an embodiment of the present disclosure;

[0066] Figure 4 This is one of the flowcharts of image-based behavior recognition according to one embodiment of the present disclosure;

[0067] Figure 5 A schematic diagram of a process of training a first neural network according to an embodiment of the present disclosure;

[0068] Figure 6 The second flowchart of image-based behavior recognition according to an embodiment of the present disclosure;

[0069] Figure 7 An image-based behavior recognition device according to an embodiment of the present disclosure;

[0070] Figure 8 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0071] To further illustrate the technical solution provided by the embodiments of the present disclosure, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present disclosure provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present disclosure. In the actual processing process or when the control device is executed, the method can be executed in the order of the method shown in the embodiments or drawings or in parallel.

[0072] The term "multiple" in the embodiments of the present disclosure refers to two or more than two, and other quantifiers are similar and should be understood. The preferred embodiments described herein are only used to illustrate and explain the present disclosure and are not used to limit the present disclosure. In addition, the embodiments in the present disclosure and the features in the embodiments may be combined with each other if there is no conflict.

[0073] The inventors have found that in existing behavior recognition methods, each frame of the same target in the input video stream or picture sequence is detected, which results in a long detection time. In addition, image frames with poor imaging effects will affect the final recognition results and may lead to recognition errors.

[0074] Therefore, the present disclosure proposes an image-based behavior recognition method, device, equipment and storage medium. The present disclosure is described in detail below in conjunction with the accompanying drawings.

[0075] Figure 1 Schematic diagram of a scenario of image-based behavior recognition, wherein the application scenario includes multiple terminal devices 110 and a server 120. Figure 1 In the figure, three terminal devices 110 are used as an example, but the number of terminal devices 110 is not limited. The terminal devices 110 and the server 120 can communicate with each other through a communication network. The terminal devices 110 are, for example, mobile phones, tablet computers, and personal computers. The server 120 can be implemented by a single server or multiple servers. The server 120 can be implemented by a physical server or a virtual server.

[0076] In a possible application scenario, the server 120 performs target detection on each frame image in the video to be detected to obtain image features of the target object; then, based on the obtained image features of the target object, the image quality of each frame image is determined. Finally, the server 120 screens out images whose image quality meets preset quality requirements, performs behavior recognition on the target object, obtains a final behavior recognition result, and sends the final behavior recognition result to the terminal device 110 for display.

[0077] In the embodiment of the present application, the image quality of the images in the input video stream or picture sequence can be screened first, and the images that meet the image quality can be used for image recognition. The selection rules can be determined according to the specific actual situation, and can include two ideas: Figure 2 As shown, Figure 2 To select images of suitable quality from videos within a specified time period for image recognition, where t represents the specified time period. Figure 3 To select images that meet the quality requirements from a video with a specified number of frames for image recognition. The video to be detected in the present disclosure may be a video within a specified time period or a video with a specified number of frames, and the present disclosure is not limited thereto. The present disclosure is described in detail below in conjunction with the accompanying drawings.

[0078] like Figure 4 As shown, Figure 4 The flowchart of the image-based behavior recognition method may include the following steps:

[0079] Step 401: performing target detection on each frame image in the video to be detected to obtain image features of the target object;

[0080] Step 402: determining the image quality of each frame of image according to the image features of the target object;

[0081] Step 403: Screen out the images whose image quality meets the preset quality requirement, and perform behavior recognition on the target object.

[0082] Therefore, the present disclosure performs target detection on each frame image in the video to obtain the image features of the target object, and then determines the image quality of each frame image through the image features, and screens out images that meet the preset quality requirements to perform behavior recognition. Therefore, it is achieved that the images that need to be recognized for behavior are recognized using a complex behavior recognition method, while filtering out some images that do not need to be recognized to improve the recognition efficiency. In addition, by performing behavior recognition on high-quality images, it is possible to avoid the problem that low-quality images are prone to behavior recognition errors that adversely affect the overall behavior recognition results.

[0083] The image features mentioned above include the deflection angle of the target object and the image clarity of the target object, wherein the deflection angle of the target object is used to identify the angle between the facial orientation of the target object and the lens plane of the camera; for example, if the angle between the facial orientation of the target object and the lens of the camera is 60 degrees, then the deflection angle of the target object is determined to be 60 degrees.

[0084] In one embodiment, step 402 can be specifically implemented as follows: according to a preset image quality scoring rule, determine the image quality score of each frame image under the deflection angle of the target object and the image clarity of the target object, and the image quality score is used to identify the image quality of the image when performing behavior recognition.

[0085] For example, according to the preset scoring rules, the image quality score of the first frame image is determined to be A, the image quality score of the second frame image is determined to be B, and the image quality of the third frame image is determined to be C. If A>B>C, then it is determined that the image quality of the first frame image>the image quality of the second frame image>the image quality of the third frame image.

[0086] Therefore, the image quality of each frame image can be scored according to the preset scoring rules to identify the quality of each frame image, so as to filter out high-quality images for behavior recognition, thereby improving the recognition accuracy.

[0087] In one embodiment, the image quality score can be determined by weighted summing the weight of the deflection angle of the target object and the weight of the image clarity of the target object. The image quality score of each frame of graphics can be calculated according to formula (1):

[0088] S = A*x+B*y(1);

[0089] Among them, S represents the score of image quality, A represents the weight of the deflection angle of the target object, B represents the weight of the image clarity of the target object, x represents the deflection angle of the target object, and y represents the image clarity.

[0090] It should be noted that the weight of the deflection angle of the target object and the weight of the image clarity of the target object can be set by empirical values, and a user interface can be provided to set and modify the weights during use. In another embodiment, each weight can be obtained by training the first neural network.

[0091] like Figure 5 As shown, Figure 5A flowchart for training a first neural network may include the following steps: Step 501: obtaining a first image feature training sample, wherein the first image feature training sample includes a deflection angle of a target object of each frame image, an image clarity of the target object, an initial weight of the deflection angle of the target object, an initial weight of the image clarity of the target object, and a preset score of each frame image marked in advance;

[0092] Step 502: inputting the first image feature training sample into the first neural network, scoring each frame of the image according to the deflection angle of the target object of each frame of the image and the image clarity of the target object, and obtaining the image quality score of each frame of the image predicted by the first neural network;

[0093] Step 503: Compare the obtained image quality score of each frame of image with the preset score of each frame of image to obtain a loss value;

[0094] Step 504: Train the first neural network according to the loss value to obtain the weight of the deflection angle of the target object and the weight of the image clarity.

[0095] Therefore, by training the first neural network, the weight of the deflection angle of the target object and the weight of the image clarity of the target object are obtained, so that the image quality score is learned based on a large number of real and reliable samples, making the weight setting more accurate.

[0096] Before executing step 402, it is necessary to first determine whether the target object is a moving target. In one embodiment, the target object is tracked according to the video to be detected to obtain a motion trajectory of the target object; if the target object is determined to be a moving target according to the motion trajectory, step 402 is executed. The obtained motion trajectory can be output so that the user can view the historical performance of the target object.

[0097] It should be noted that when it is determined from the motion trajectory that the target object is not a moving target, there is no need to perform step 402. Because when it is determined that the target object is not a moving target, it can be determined that the target object may be detected incorrectly and target detection needs to be performed again.

[0098] Therefore, only the determined moving target needs to have its image quality determined, and it is not necessary to determine the image quality of each frame of image. Therefore, the amount of calculation is saved and the accuracy of the recognition result is improved.

[0099] The present disclosure can also save computational effort through filtering. In one embodiment, for any frame of image, if the confidence level of the deflection angle of the target object in the image and the confidence level of the image sharpness of the target object do not meet the specified conditions, the image is filtered; the specified conditions include that the confidence level of the deflection angle of the target object is greater than a first preset threshold and the confidence level of the image sharpness of the target object is greater than a second preset threshold.

[0100] For example, the first preset threshold is M, the second preset threshold is N, the confidence level of the deflection angle of the target object is m, and the confidence level of the image sharpness is n. When m > M and n > N, step 402 can be executed. For any frame of image, if the above specified conditions are not met, that is, m < M and / or n < N, then the frame of image is filtered, that is, the quality of the frame of image does not need to be determined.

[0101] Thus, when one or more frames of images determined do not meet the conditions, the one or more frames of images are filtered. This is used to save computational effort.

[0102] In one embodiment, the image feature further includes contour information. The aforementioned step 403 can be implemented as:

[0103] Obtain the contour information of the target object in the selected image; and input the selected image into a second neural network for behavior recognition to obtain behavior features; and,

[0104] Input the selected image into a third neural network for face recognition to obtain face attribute features;

[0105] Match the behavior feature matrix obtained by combining the face attribute features and the behavior features with a pre-annotated behavior feature matrix to obtain a final behavior recognition result.

[0106] It should be noted that the behavior features can be the limb features of the target object. The face attribute features are the angles of the target object's head with respect to the horizontal direction. For example, the angle of looking up, the angle of looking down, and looking straight ahead.

[0107] For example, when recognizing the behavior of a student, the obtained behavior feature is sitting, and the obtained face attribute feature is looking straight ahead. The behavior feature matrix obtained by combining the face attribute features and the behavior features is matched with a pre-annotated behavior feature matrix, and the final behavior recognition result of reading can be obtained.

[0108] Thus, by using the behavior features obtained by behavior recognition and combining the face attribute features obtained by face recognition to obtain the final behavior recognition result, the recognition accuracy can be further improved.

[0109] In order to further understand the technical solution of the present disclosure, Figure 6 A detailed description may include the following steps:

[0110] Step 601: Perform target detection on each frame image in the video to be detected to obtain image features of the target object; the image features include the deflection angle of the target object, the image clarity and contour information of the target object;

[0111] The deflection angle of the target object is used to identify the angle between the facial orientation of the target object and the lens plane of the camera.

[0112] Step 602: the image feature further includes a confidence level; for any frame of image, if the confidence level of the deflection angle of the target object of the image and the confidence level of the image clarity of the target object do not meet a specified condition, the image is filtered; the specified condition includes that the confidence level of the deflection angle of the target object is greater than a first preset threshold and the confidence level of the image clarity of the target object is greater than a second preset threshold;

[0113] Step 603: performing weighted summation according to the weight of the deflection angle of the target object and the weight of the image clarity of the target object to obtain the image quality score;

[0114] Step 603: Screening out images whose image quality meets a preset quality requirement;

[0115] Step 604: Acquire contour information of the target object in the filtered image; and input the filtered image into a second neural network for behavior recognition to obtain behavior features;

[0116] Step 605: inputting the screened image into a third neural network for face recognition to obtain facial attribute features;

[0117] It should be noted that the execution order of step 605 and step 606 is not limited here.

[0118] Step 606: Matching a behavior feature matrix obtained by combining the facial attribute features with the behavior features with a pre-labeled behavior feature matrix to obtain a final behavior recognition result.

[0119] Based on the same inventive concept, the image-based behavior recognition method described above in the present disclosure can also be implemented by an image-based behavior recognition device. The effect of the device is similar to that of the aforementioned method, and will not be described in detail here.

[0120] Figure 7 The figure is a schematic diagram of the structure of an image-based behavior recognition method according to an embodiment of the present disclosure.

[0121] like Figure 7 As shown, the image-based behavior recognition device 700 of the present disclosure may include a target detection module 710 , an image quality determination module 720 , and a behavior recognition module 730 .

[0122] The target detection module 710 is used to perform target detection on each frame image in the video to be detected to obtain image features of the target object;

[0123] An image quality determination module 720, configured to determine the image quality of each frame of image according to the image features of the target object;

[0124] The behavior recognition module 730 is used to screen out the images whose image quality meets the preset quality requirements and perform behavior recognition on the target object.

[0125] In one embodiment, the image features include the deflection angle of the target object and the image clarity of the target object; wherein,

[0126] The deflection angle of the target object is used to identify the angle between the facial orientation of the target object and the lens plane of the camera;

[0127] The image quality determination module 720 is further configured to:

[0128] According to a preset image quality scoring rule, the image quality score of each frame of image under the deflection angle of the target object and the image clarity of the target object is determined.

[0129] In one embodiment, the image quality determination module 720 is further configured to:

[0130] Performing a weighted sum according to the weight of the deflection angle of the target object and the weight of the image clarity of the target object to obtain a score of the image quality;

[0131] The weight of the deflection angle of the target object and the weight of the image definition of the target object are obtained by training a first neural network, and the device further includes:

[0132] The training sample acquisition module 740 is used to train the first neural network by the following method: acquiring a first image feature training sample, wherein the first image feature training sample includes a deflection angle of a target object of each frame image, an image clarity of the target object, an initial weight of the deflection angle of the target object, an initial weight of the image clarity of the target object, and a preset score of each frame image marked in advance;

[0133] A scoring module 750 is used to input the first image feature training sample into the first neural network, score each frame of the image according to the deflection angle of the target object of each frame of the image and the image clarity of the target object, and obtain a score of the image quality of each frame of the image predicted by the first neural network;

[0134] A comparison module 760, configured to compare the obtained image quality score of each frame of image with a preset score of each frame of image to obtain a loss value;

[0135] The weight acquisition module 770 is used to train the first neural network according to the loss value to obtain the weight of the deflection angle of the target object and the weight of the image clarity.

[0136] In one embodiment, the apparatus further comprises:

[0137] A tracking module 780 is used to track the target object according to the video to be detected to obtain a motion trajectory of the target object before determining the image quality of each frame of the image according to the image features of the target object;

[0138] The moving target determination module 790 is used to execute the step of determining the image quality of each frame image according to the image features of the target object if the target object is determined to be a moving target according to the motion trajectory.

[0139] In one embodiment, the image feature further includes a confidence level; and the device further includes:

[0140] The filtering module 791 is used to filter the image for any frame image before determining the image quality of each frame image according to the image features of the target object, if the confidence of the deflection angle of the target object and the confidence of the image clarity of the target object do not meet specified conditions; the specified conditions include that the confidence of the deflection angle of the target object is greater than a first preset threshold and the confidence of the image clarity of the target object is greater than a second preset threshold.

[0141] In one embodiment, the image features further include contour information;

[0142] The behavior recognition module 730 is further used for:

[0143] Acquire the contour information of the target object in the screened image; input the screened image into a second neural network for behavior recognition to obtain behavior features; and,

[0144] Inputting the screened image into a third neural network for face recognition to obtain facial attribute features;

[0145] The behavior feature matrix obtained by combining the facial attribute features with the behavior features is matched with a pre-labeled behavior feature matrix to obtain a final behavior recognition result.

[0146] After introducing an image-based behavior recognition method according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0147] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0148] In some possible implementations, the electronic device according to the present application may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the image-based behavior recognition method according to various exemplary implementations of the present application described above in this specification. For example, the processor may execute the following steps: Figure 4 Steps 401-403 shown in .

[0149] Refer to the following Figure 8 hereinafter describes an electronic device 800 according to this embodiment of the present application. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0150] like Figure 8 As shown, the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include but are not limited to: at least one processor 801, at least one computer storage medium 802, and a bus 803 connecting different system components (including the computer storage medium 802 and the processor 801).

[0151] Bus 803 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0152] The computer storage medium 802 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 821 and / or cache storage media 822 , and may further include read-only computer storage media (ROM) 823 .

[0153] The computer storage medium 802 may also include a program / utility 825 having a set (at least one) of program modules 824, such program modules 824 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0154] The electronic device 800 can also communicate with one or more external devices 805 (e.g., keyboards, pointing devices, etc.), can also communicate with one or more devices that enable users to interact with the electronic device 800, and / or communicate with any device (e.g., routers, modems, etc.) that enables the electronic device 800 to communicate with one or more other electronic devices. This communication can be carried out through an input / output (I / O) interface 805. In addition, the electronic device 800 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter 806. As shown in the figure, the network adapter 806 communicates with other modules for the electronic device 800 through a bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0155] In some possible implementations, various aspects of an image-based behavior recognition method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of an image-based behavior recognition method according to various exemplary implementations of the present application described above in this specification.

[0156] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access computer storage medium (RAM), a read-only computer storage medium (ROM), an erasable programmable read-only computer storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only computer storage medium (CD-ROM), an optical computer storage medium, a magnetic computer storage medium, or any suitable combination thereof.

[0157] The program product for image-based behavior recognition of the embodiment of the present application can adopt a portable compact disk read-only computer storage medium (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0158] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0159] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0160] The program code for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and also conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or completely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (e.g., using an Internet service provider to connect through the Internet).

[0161] It should be noted that, although several modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0162] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk computer storage media, CD-ROM, optical computer storage media, etc.) that contain computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable computer storage medium produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image-based behavior recognition method, characterized in that: The method comprises: Performing target detection on each frame image in the video to be detected to obtain image features of the target object, wherein the image features include a deflection angle of the target object and image clarity of the target object, and the deflection angle of the target object is used to identify the angle between the facial orientation of the target object and the lens plane of the camera; Determining the image quality of each frame of image according to the image feature of the target object specifically includes: The image quality score is obtained by performing weighted summation according to the weight of the deflection angle of the target object and the weight of the image clarity of the target object, wherein the weight of the deflection angle of the target object and the weight of the image clarity of the target object are obtained by training a first neural network, and the first neural network is trained by the following method: obtaining a first image feature training sample, wherein the first image feature training sample includes the deflection angle of the target object of each frame image, the image clarity of the target object, the initial weight of the deflection angle of the target object, the initial weight of the image clarity of the target object, and the preset score of each frame image marked in advance; inputting the first image feature training sample into the first neural network, scoring each frame image according to the deflection angle of the target object of each frame image and the image clarity of the target object, and obtaining the image quality score of each frame image predicted by the first neural network; comparing the obtained image quality score of each frame image with the preset score of each frame image to obtain a loss value; training the first neural network according to the loss value to obtain the weight of the deflection angle of the target object and the weight of the image clarity; The images whose image quality meets the preset quality requirement are screened out, and behavior recognition for the target object is performed.

2. The method according to claim 1, characterized in that Before determining the image quality of each frame of image according to the image feature of the target object, the method further includes: Tracking the target object according to the video to be detected to obtain a motion trajectory of the target object; If the target object is determined to be a moving target according to the motion trajectory, a step of determining the image quality of each frame image according to the image feature of the target object is performed.

3. The method according to claim 1, characterized in that The image features also include confidence; Before determining the image quality of each frame of image according to the image feature of the target object, the method further includes: For any frame image, if the confidence of the deflection angle of the target object of the image and the confidence of the image clarity of the target object do not meet specified conditions, the image is filtered; the specified conditions include that the confidence of the deflection angle of the target object is greater than a first preset threshold and the confidence of the image clarity of the target object is greater than a second preset threshold.

4. The method according to claim 1, characterized in that The image features also include contour information; The step of screening out the images whose image quality meets the preset quality requirement and performing behavior recognition on the target object includes: Acquire the contour information of the target object in the screened image; input the screened image into a second neural network for behavior recognition to obtain behavior features; and, Inputting the screened image into a third neural network for face recognition to obtain facial attribute features; The behavior feature matrix obtained by combining the facial attribute features with the behavior features is matched with a pre-labeled behavior feature matrix to obtain a final behavior recognition result.

5. An image-based behavior recognition device, characterized in that: The device comprises: A target detection module, used to perform target detection on each frame image in the video to be detected, and obtain image features of the target object, wherein the image features include a deflection angle of the target object and an image clarity of the target object, and the deflection angle of the target object is used to identify the angle between the facial orientation of the target object and the lens plane of the camera; An image quality determination module, used to determine the image quality of each frame of the image according to the image features of the target object, wherein the image quality determination module is further used to: perform weighted summation according to the weight of the deflection angle of the target object and the weight of the image clarity of the target object to obtain the image quality score, wherein the weight of the deflection angle of the target object and the weight of the image clarity of the target object are obtained by training a first neural network; A training sample acquisition module is used to train the first neural network by the following method: acquiring a first image feature training sample, wherein the first image feature training sample includes a deflection angle of a target object of each frame image, an image clarity of the target object, an initial weight of the deflection angle of the target object, an initial weight of the image clarity of the target object, and a preset score of each frame image marked in advance; a scoring module, configured to input the first image feature training sample into the first neural network, score each frame of the image according to the deflection angle of the target object of each frame of the image and the image clarity of the target object, and obtain a score of the image quality of each frame of the image predicted by the first neural network; A comparison module, used for comparing the obtained image quality score of each frame of image with the preset score of each frame of image to obtain a loss value; A weight acquisition module, used for training the first neural network according to the loss value to obtain the weight of the deflection angle of the target object and the weight of the image clarity; The behavior recognition module is used to screen out the images whose image quality meets the preset quality requirements and perform behavior recognition on the target object.

6. The device according to claim 5, characterized in that The device also includes: A tracking module, used for tracking the target object according to the video to be detected to obtain a motion trajectory of the target object before determining the image quality of each frame of the image according to the image features of the target object; The moving target determination module is used to execute the step of determining the image quality of each frame image according to the image features of the target object if the target object is determined to be a moving target according to the motion trajectory.

7. The device according to claim 5, characterized in that The image feature also includes a confidence level; and the device also includes: A filtering module is used to filter the image before determining the image quality of each frame of the image based on the image features of the target object. For any frame of the image, if the confidence of the deflection angle of the target object and the confidence of the image clarity of the target object do not meet specified conditions, the specified conditions include that the confidence of the deflection angle of the target object is greater than a first preset threshold and the confidence of the image clarity of the target object is greater than a second preset threshold.

8. The device according to claim 5, characterized in that The image features also include contour information; The behavior recognition module is further used for: Acquiring contour information of the target object in the screened image; and inputting the screened image into a second neural network for behavior recognition to obtain behavior features; and, Inputting the screened image into a third neural network for face recognition to obtain facial attribute features; The behavior feature matrix obtained by combining the facial attribute features with the behavior features is matched with a pre-labeled behavior feature matrix to obtain a final behavior recognition result.

9. An electronic device, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor; and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-4.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1-4.