Behavior recognition method and device, equipment and medium

Abstract

The invention discloses a behavior recognition method and device, equipment and a medium, and relates to the technical field of behavior recognition. The method comprises the steps of obtaining a to-be-recognized image set based on to-be-recognized images obtained by imaging a target object in a preset small-span duration by at least two image sensors; and inputting the to-be-recognized image set into the trained behavior recognition network to obtain a set behavior recognition result output by the behavior recognition network. The behavior recognition accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of behavior recognition technology, and in particular to a behavior recognition method, device, equipment and medium. Background Art

[0002] In the fields of video surveillance, video recommendation, and human-computer interaction, behavior recognition can identify the actions of people in the captured images or video frames. For example, in the field of transportation, behavior recognition can be used to determine whether the driver is driving or talking on the phone while driving.

[0003] However, in the related art, the recognition accuracy of behavior recognition based on a single image is low, while the recognition accuracy of behavior recognition based on a video is easily affected by the changes in the behavior in the video and thus reduced. Summary of the invention

[0004] The main purpose of this application is to provide a behavior recognition method, device, equipment and medium, aiming to solve the technical problem of low behavior recognition accuracy.

[0005] To achieve the above objectives, the present application provides a behavior recognition method, the method comprising:

[0006] A set of images to be identified is obtained based on images to be identified obtained by imaging the target object by at least two image sensors within a preset short span time; wherein at least two image sensors among all the image sensors have different working bands and / or spatial positions;

[0007] The set of images to be identified is input into the trained behavior recognition network to obtain the collective behavior recognition result output by the behavior recognition network; wherein the behavior recognition network includes a feature extraction module, a first behavior recognition module, a recognition quality assessment module and a second behavior recognition module; the feature extraction module is used to extract features from each image to be identified to obtain image features; the first behavior recognition module is used to perform behavior recognition on each image to be identified based on the image features to obtain a predicted behavior recognition result of each image to be identified; the recognition quality assessment module is used to perform recognition quality assessment on each image to be identified to obtain a recognition weight of each image to be identified; the second behavior recognition module is used to obtain a collective behavior recognition result based on the predicted behavior recognition result and the corresponding recognition weight.

[0008] In a possible embodiment of the present application, obtaining a set of images to be identified based on images to be identified obtained by imaging a target object by at least two image sensors respectively includes:

[0009] Acquire multiple frames of real-time images collected by the image sensor at different times within a preset short span time;

[0010] Perform feature recognition and feature clustering on multiple frames of real-time images to obtain feature clustering results;

[0011] Based on the feature clustering results, the images to be identified including the target object are screened out from multiple frames of real-time images;

[0012] Based on the images to be recognized from at least two image sensors, a set of images to be recognized is obtained.

[0013] In a possible embodiment of the present application, after inputting the set of images to be recognized into a trained behavior recognition network and obtaining the set behavior recognition result output by the behavior recognition network, the method further includes:

[0014] Output the recognition weight of each image to be recognized;

[0015] Based on all identification weights, unqualified points are screened out.

[0016] In a possible embodiment of the present application, unqualified points are screened out based on all recognition weights, including:

[0017] Determine all identified target objects targeted by the image sensor within a preset working time, and the existing recognition weights corresponding to each identified target object; the preset working time includes at least two preset small span time lengths;

[0018] Screening out low-quality recognition weights from all existing recognition weights; wherein the low-quality recognition weight is less than the preset recognition weight;

[0019] Determine the proportion of all low-quality identification weights to all existing identification weights;

[0020] If the proportion is less than the preset warning threshold, the point where the image sensor is located is determined as an unqualified point.

[0021] In a possible embodiment of the present application, based on the images to be identified obtained by imaging the target object by at least two image sensors within a preset short span time, before obtaining the set of images to be identified, the method further includes:

[0022] Acquire at least two initial sample images; wherein the at least two initial sample images are acquired by multiple image sensors at the same time and for the same sample object;

[0023] Based on the initial sample image, an expanded sample image with similar image features is generated;

[0024] Based on the initial sample image and the expanded sample image, a training sample is obtained;

[0025] The action recognition network is trained using at least one training sample.

[0026] In a possible embodiment of the present application, training the behavior recognition network using at least one training sample includes:

[0027] Inputting the training sample into the behavior recognition network, obtaining a first predicted behavior category output by the behavior recognition network and at least two predicted recognition weights output by the recognition quality assessment module;

[0028] Constructing a first loss function based on the first predicted behavior category and the first true behavior category of the training sample;

[0029] constructing a second loss function based on at least two predicted recognition weights and a true weight corresponding to each image in the training sample;

[0030] Based on the first loss function and the second loss function, the behavior recognition network is trained.

[0031] In a possible embodiment of the present application, based on the first loss function and the second loss function, the behavior recognition network is trained, including:

[0032] Obtain a second predicted behavior category output by the first behavior recognition module for each image;

[0033] determining a recognition loss between a second predicted behavior category and a corresponding second true behavior category of the image;

[0034] Based on the recognition loss of all images, construct the third loss function;

[0035] Based on the first loss function, the second loss function and the third loss function, the behavior recognition network is trained.

[0036] In a second aspect, the present application further provides a behavior recognition device, the device comprising:

[0037] An image acquisition module, used to obtain a set of images to be identified based on images to be identified obtained by imaging the target object by at least two image sensors within a preset short span time, wherein at least two image sensors among all the image sensors have different working bands and / or spatial positions;

[0038] The network recognition module is used to input the set of images to be recognized into the trained behavior recognition network to obtain the collective behavior recognition result output by the behavior recognition network; wherein the behavior recognition network includes a feature extraction module, a first behavior recognition module, a recognition quality assessment module and a second behavior recognition module; the feature extraction module is used to extract features from each image to be recognized to obtain image features; the first behavior recognition module is used to perform behavior recognition on each image to be recognized based on the image features to obtain the predicted behavior recognition result of each image to be recognized; the recognition quality assessment module is used to perform recognition quality assessment on each image to be recognized to obtain the recognition weight of each image to be recognized; the second behavior recognition module is used to obtain the collective behavior recognition result based on the predicted behavior recognition result and the corresponding recognition weight.

[0039] In a possible embodiment of the present application, the image acquisition module specifically includes:

[0040] An image acquisition unit, used to acquire multiple frames of real-time images collected by an image sensor at different times within a preset short span time;

[0041] A clustering unit, used for performing feature recognition and feature clustering on multiple frames of real-time images to obtain feature clustering results;

[0042] An image screening unit, used for screening out the image to be identified including the target object from the multiple frames of real-time images based on the feature clustering result;

[0043] The set construction unit is used to obtain a set of images to be identified based on the images to be identified from at least two image sensors.

[0044] In a possible embodiment of the present application, the behavior recognition device further includes:

[0045] A weight output module, used to output the recognition weight of each image to be recognized;

[0046] The point screening module is used to screen out unqualified points based on all recognition weights.

[0047] In a possible embodiment of the present application, the point screening module specifically includes:

[0048] A data statistics unit, used to determine all identified target objects targeted by the image sensor within a preset working time, and the existing recognition weights corresponding to each identified target object; the preset working time includes at least two preset small span time lengths;

[0049] A weight screening unit, used to screen out low-quality recognition weights from all existing recognition weights; wherein the low-quality recognition weight is less than a preset recognition weight;

[0050] A proportion calculation unit, used to determine the proportion of all low-quality recognition weights in all existing recognition weights;

[0051] The point screening unit is used to determine the point where the image sensor is located as an unqualified point if the proportion is less than a preset warning threshold.

[0052] In a possible embodiment of the present application, the behavior recognition device further includes:

[0053] A sample image acquisition module, used to acquire at least two initial sample images; wherein the at least two initial sample images are acquired by multiple image sensors at the same time and for the same sample object;

[0054] An image expansion module, used for generating an expanded sample image with similar image features based on an initial sample image;

[0055] A sample construction module, used to obtain training samples based on the initial sample image and the expanded sample image;

[0056] The training module is used to train the behavior recognition network using at least one training sample.

[0057] In a possible embodiment of the present application, the training module is specifically used to: input the training sample into the behavior recognition network to obtain the first predicted behavior category output by the behavior recognition network and at least two predicted recognition weights output by the recognition quality assessment module; construct a first loss function based on the first predicted behavior category and the first real behavior category of the training sample; construct a second loss function based on the at least two predicted recognition weights and the real weights corresponding to each image in the training sample; and train the behavior recognition network based on the first loss function and the second loss function.

[0058] In a possible embodiment of the present application, the training module is further specifically used to obtain the second predicted behavior category output by the first behavior recognition module for each image; determine the recognition loss between the second predicted behavior category and the second real behavior category of the corresponding image; construct a third loss function based on the recognition loss of all images; and train the behavior recognition network based on the first loss function, the second loss function and the third loss function.

[0059] In a third aspect, the present application further provides a behavior recognition device, including: a processor, a memory, and a computer program stored in the memory, wherein the computer program implements the behavior recognition method of the first aspect when executed by the processor.

[0060] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the behavior recognition method of the first aspect is implemented.

[0061] A behavior recognition method proposed in an embodiment of the present application includes: obtaining a set of images to be recognized based on images to be recognized obtained by imaging a target object respectively by at least two image sensors within a preset short span time; wherein, the working bands and / or spatial positions of at least two image sensors among all the image sensors are different; inputting the set of images to be recognized into a trained behavior recognition network to obtain a collective behavior recognition result output by the behavior recognition network; wherein, the behavior recognition network includes a feature extraction module, a first behavior recognition module, a recognition quality assessment module and a second behavior recognition module; the feature extraction module is used to extract features from each image to be recognized to obtain image features; the first behavior recognition module is used to perform behavior recognition on each image to be recognized based on the image features to obtain a predicted behavior recognition result of each image to be recognized; the recognition quality assessment module is used to perform recognition quality assessment on each image to be recognized to obtain a recognition weight of each image to be recognized; the second behavior recognition module is used to obtain a collective behavior recognition result based on the predicted behavior recognition result and the corresponding recognition weight.

[0062] It can be seen that compared with behavior recognition based on a single image, the object of behavior recognition in the embodiment of the present application is multiple images to be recognized obtained by imaging with image sensors in multiple working bands or different spatial positions, and the recognition quality assessment module in the behavior recognition network is used to perform recognition quality assessment to obtain corresponding recognition weights, so as to supervise the correlation between the multiple images to be recognized. In this way, while expanding the image source, more images with better recognition quality are considered for behavior recognition, thereby improving the accuracy of behavior recognition.

[0063] Compared with behavior recognition on videos, the images to be recognized in the set of images to be recognized for behavior recognition in the embodiment of the present application can be images collected separately for the same target object by image sensors in different working bands or different spatial positions within a preset small span time. As a result, there is no time constraint between the multiple images to be recognized, thus avoiding the influence of behavior changes in the video. Different working bands or different spatial positions can provide more image features for behavior recognition, thereby improving the accuracy of behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a schematic diagram of the structure of the behavior recognition device of this application;

[0065] Figure 2 This is a flow chart of the first embodiment of the behavior recognition method of the present application;

[0066] Figure 3 This is a schematic diagram of the structure of the behavior recognition network of this application;

[0067] Figure 4 This is a flow chart of the second embodiment of the behavior recognition method of the present application;

[0068] Figure 5 This is a flow chart of the third embodiment of the behavior recognition method of the present application;

[0069] Figure 6 This is a module diagram of the behavior recognition device of this application.

[0070] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0071] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] In the fields of video surveillance, video recommendation, and human-computer interaction, behavior recognition can identify the actions of people in the captured images or video frames. For example, in the field of transportation, behavior recognition can be used to determine whether the driver is driving or talking on the phone while driving.

[0073] However, in the related art, the recognition accuracy of behavior recognition based on a single image is low, and the recognition accuracy of behavior recognition based on a video is easily affected by the changes in the behavior in the video and is also reduced.

[0074] To this end, the embodiment of the present application provides a solution, which uses images collected by image sensors at multiple different spatial positions and / or different working bands to obtain multiple images to be identified. Compared with a single image, this embodiment expands the image source while considering more images with better recognition quality for behavior recognition, thereby improving the accuracy of behavior recognition. Compared with video, there is no time constraint between the multiple images to be identified in this embodiment, avoiding the influence of behavior changes in the video, and different working bands or different spatial positions can provide more image features for behavior recognition, thereby improving the accuracy of behavior recognition.

[0075] The following is a description of the technologies involved in the embodiments of the present application.

[0076] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0077] The present application relates to computer vision technology (CV) in artificial intelligence. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify, record and measure targets, and further perform graphic processing so that the computer processes the images into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Specifically, in the embodiments of the present application, it mainly relates to behavior recognition in computer vision technology.

[0078] The inventive concept of the embodiments of the present application is further explained below in conjunction with some specific embodiments and implementation methods.

[0079] Reference Figure 1 , Figure 1 A schematic diagram of the structure of a behavior recognition device in a hardware operating environment involved in an embodiment of the present application.

[0080] like Figure 1As shown, the behavior recognition device may include: a processor 1001, such as a CPU, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a voice pickup module, such as a microphone array, etc., and the optional user interface 1003 may also be a display screen (Display), an input unit such as a keyboard (Keyboard), etc. The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk storage. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0081] It is understandable that the behavior recognition device may also include a network interface 1004, which may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). Optionally, the behavior recognition device may also include an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. The behavior recognition device may also include a display screen, which is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen is a touch display screen, the display screen also has the ability to collect touch signals on the surface or above the surface of the display screen. The touch signal may be input to the processor 301 as a control signal for processing.

[0082] Those skilled in the art will understand that Figure 1 The behavior recognition device structure shown in the figure does not constitute a limitation on the behavior recognition device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0083] Based on the above structure but not limited to the above structure, the first embodiment of the behavior recognition method of the present application is proposed. Figure 2 , Figure 2 This is a flow chart of the first embodiment of the behavior recognition method of the present application.

[0084] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0085] In this embodiment, the behavior recognition method includes:

[0086] Step S100: obtaining a set of images to be identified based on images to be identified obtained by imaging a target object by at least two image sensors within a preset short span time.

[0087] Among them, at least two image sensors among all the image sensors have different working bands and / or different spatial positions.

[0088] Specifically, the executor of the method in this embodiment is a behavior recognition device. The behavior recognition device can be a front-end device such as an image acquisition device, that is, it has both an image acquisition function and a behavior recognition function, so that after acquiring a real-time image of the target object, the real-time image is locally identified by the device. Alternatively, the behavior recognition device can also be a back-end device. After the front-end device acquires the real-time image of the target object and transmits it to the back-end device through the network, the back-end device performs behavior recognition on the real-time image. It can be understood that the back-end device can be a server, a cloud server, or a virtual cluster deployed in the cloud, etc.

[0089] The target object is the identification object targeted by the behavior recognition in this embodiment. Of course, the target object can be a living being such as a person or animal or a non-living being such as a robot that can perform corresponding behaviors or actions. Taking the field of transportation as an example, the target object of behavior recognition can be the driver of a vehicle traveling on the road, so as to perform behavior recognition on the driver and determine whether his behavior is normal driving behavior or illegal driving behavior that violates traffic laws and regulations.

[0090] The image sensor is used to convert the optical signal into an electrical signal, so as to image the target object. It is understandable that in this embodiment, a plurality of image sensors with different working bands can be arranged on the same image acquisition device, so as to acquire multiple frames of real-time images of the target object at the same time, that is, to obtain multiple frames of images to be recognized. Alternatively, a plurality of image sensors with different working bands can also be arranged on image acquisition devices at different spatial positions, so that the target object can be imaged at different positions in space at the same time or within a preset small span time, so as to obtain multiple frames of images to be recognized. It is worth mentioning that in this embodiment, the image sensor can be a visible light image sensor, a near-infrared image sensor, a microwave image sensor, an infrared image sensor, etc.

[0091] For example, in the field of transportation, the multiple image sensors may be multiple image sensors with different working bands arranged at the same station on the road, such as infrared image sensors, visible light infrared sensors, etc. Alternatively, the multiple image sensors may be multiple image sensors with the same working band or different working bands arranged in sequence along the extension direction of a designated road section. Of course, the multiple image sensors may also include multiple image sensors with different working bands arranged at the same station on the road, and multiple image sensors with the same working band or different working bands arranged in sequence along the extension direction of a designated road section.

[0092] Multiple image sensors respectively image the target object within a preset small span duration to be recognized. It is worth mentioning that the preset small span duration can be at the same moment, or it can be within adjacent moments, or the duration of the preset small span duration should be sufficient to satisfy the time span so that the behavior of the target object has not changed. Of course, the specific duration of the preset small span duration can be determined according to the application field of behavior recognition and / or the specific action scope of the target object. For example, in the field of human-computer interaction, the behavior recognized by behavior recognition is generally a hand gesture, so the preset small span duration can be at adjacent moments, or the duration of the preset small span duration is less than 3S. In the field of traffic, the behavior recognized by behavior recognition is whether the driver is making a phone call, smoking, etc., so the duration of the preset small span duration can be 1s-10s. As a result, a single image sensor can obtain only one frame of image to be recognized, or multiple frames of image to be recognized.

[0093] After the image sensor images the image to be recognized, the behavior recognition device can obtain at least two frames of the image to be recognized, thereby constructing a set of images to be recognized. It can be seen that the multiple frames of images to be recognized in the set of images to be recognized are different from the temporally continuous video frames in the video. They can be images at the same time or at different times, and there is no time sequence requirement for the images at different times.

[0094] In some specific implementations, step S100 specifically includes:

[0095] Step S110: acquiring multiple frames of real-time images collected by the image sensor at different times within a preset short span time.

[0096] Step S120: performing feature recognition and feature clustering on multiple frames of real-time images to obtain feature clustering results.

[0097] Step S130: based on the feature clustering result, select the image to be identified including the target object from the multiple frames of real-time images.

[0098] Step S140: obtaining a set of images to be recognized based on the images to be recognized from at least two image sensors.

[0099] Specifically, in actual application, the images to be identified obtained by imaging the image sensor may not belong to the same target object. Taking user verification in human-computer interaction as an example, it is possible that during the action verification process, two people make actions within the imaging range of the image sensor. Or taking driver behavior recognition in the field of transportation as an example, within a certain demarcated road section, the driver of the vehicle may have changed. At this time, in order to ensure that all behaviors in the set of images to be identified are made by the same target object, it is necessary to perform feature recognition and feature clustering on multiple frames of real-time images to obtain feature clustering results, so as to determine whether they belong to the same target object based on the feature clustering results, that is, the feature distance of the features on each frame image. In this way, the images to be identified belonging to the same target object are screened out, and then the set of images to be identified is constructed. Among them, the features in this embodiment can be physical features such as the face, skull or nose of the human body, or can also be posture features such as the sitting posture of the human body, and the above feature recognitions are all authorized by the user.

[0100] Step S200: input the set of images to be recognized into the trained behavior recognition network to obtain the set behavior recognition result output by the behavior recognition network.

[0101] Among them, the behavior recognition network includes a feature extraction module, a first behavior recognition module, a recognition quality assessment module and a second behavior recognition module. The feature extraction module is used to extract features from each image to be recognized and obtain image features. The first behavior recognition module is used to perform behavior recognition on each image to be recognized based on image features to obtain predicted behavior recognition results for each image to be recognized. The recognition quality assessment module is used to perform recognition quality assessment on each image to be recognized and obtain recognition weights for each image to be recognized. The second behavior recognition module is used to obtain a collective behavior recognition result based on the predicted behavior recognition result and the corresponding recognition weight.

[0102] Specifically, see Figure 3 In this embodiment, the behavior recognition network is constructed as a dual-branch network, and the backbone network of the dual-branch network is a feature extraction module, which can be constructed as a CNN (Convolutional Neural Networks) structure or a transformer neural network structure. The input of the backbone network is a set of images to be recognized, and the output is the image features of each image to be recognized.

[0103] It is worth mentioning that in some specific embodiments, the backbone network also includes a preprocessing module, which is used to preprocess each image to be identified. The specific preprocessing step can be to perform target detection on the image to be identified, so as to generate a target frame on the image to be identified. For example, in the field of traffic, human body detection can be performed to generate a human body frame. It can be understood that for each target, there are multiple target frames generated, so NMS (Non-Maximum Suppression) processing is also required, so that in the image to be identified, only one target frame is retained for each target. Then the preprocessing module performs an external expansion processing on the target frame according to the preset external expansion size, and then cuts out the target object area through the cutout processing. The cutout processing here is to remove other areas that are not related to the target object in the image to be identified, and only retain the target object area with the target object. The subsequent behavior recognition only performs behavior recognition on the target object area, thereby reducing the amount of calculation. The external expansion processing is to avoid missing the image features of the target object in the cutout processing.

[0104] Furthermore, in some embodiments, the preprocessing step also includes an alignment process, that is, resizing the target object area obtained by the cutout process according to a specified size, thereby obtaining an image set, and inputting the image set into the feature extraction module.

[0105] The main network is connected to the first branch network and the second branch network. The first branch network includes a first behavior recognition module, that is, the main network sequentially inputs the image features extracted from each image to be recognized into the first behavior recognition module of the first branch network, and the first behavior recognition module performs behavior recognition on each image to be recognized, thereby obtaining the first vector feature of each image to be recognized, and then uses the behavior category mapped or classified by the first vector feature as the predicted behavior recognition result.

[0106] The second branch network includes a recognition quality assessment module and a second behavior recognition module, and the second behavior recognition module is connected to the first behavior recognition module and the recognition quality assessment module. Specifically, the recognition quality assessment module in this embodiment is used to measure or evaluate the difficulty and / or accuracy of image discrimination behavior categories. It can be understood that for images with high recognition quality, behavior recognition can be performed more easily and / or more accurately. On the contrary, for images with low recognition quality, behavior recognition is not easy and / or difficult to perform accurately. In this way, the recognition quality assessment module performs recognition quality assessment on each image to be recognized, thereby obtaining the recognition quality ranking result of each image to be recognized in the set of images to be recognized. The recognition quality ranking result is used to reflect the recognition quality ranking of each frame of the image to be recognized in the set of images to be recognized, so as to supervise the correlation between images in the set to be recognized, so as to allow the model to select the optimal behavior recognition image for behavior recognition.

[0107] The recognition quality ranking result can be specifically expressed as a quality score, and multiple quality scores are sorted according to the numerical value. The one with a larger numerical value is ranked first in the recognition quality, and it performs better in the behavior recognition quality. Of course, the recognition quality ranking result can also be expressed as a sorting sequence number. The image to be recognized with a smaller sorting sequence number has a better recognition quality.

[0108] In some embodiments, the recognition quality assessment module includes an assessment model and a softmax normalization layer. The image features of each image to be recognized are input into the assessment model, the assessment model outputs a corresponding quality assessment vector, and the quality assessment vector is input into the softmax normalization layer, and the softmax normalization layer outputs the recognition quality ranking result corresponding to each image to be recognized based on all the quality assessment vectors.

[0109] In the behavior recognition network of this embodiment, the recognition quality ranking result of the image to be recognized is the recognition weight.

[0110] The second behavior recognition module is used to obtain a collective behavior recognition result based on the predicted behavior recognition result and the corresponding recognition weight. In some embodiments, in order to save computing time, the first behavior recognition module may send the first vector feature to the second behavior recognition module, and the evaluation model inputs the quality assessment vector to the second behavior recognition module. The second behavior recognition module multiplies and accumulates all the first vector features and the corresponding quality assessment vector to obtain the second vector feature, and then normalizes the second vector feature to obtain the collective behavior recognition result.

[0111] It can be seen that compared with behavior recognition based on a single image, the object of behavior recognition in this embodiment is a plurality of images to be recognized obtained by imaging with image sensors in multiple working bands or at different spatial positions, and the recognition quality assessment module in the behavior recognition network performs recognition quality assessment to obtain corresponding recognition weights, so as to supervise the correlation between the multiple images to be recognized, thereby expanding the image source while considering more images with better recognition quality for behavior recognition, thereby improving the accuracy of behavior recognition.

[0112] Compared with behavior recognition on videos, the images to be recognized in the set of images to be recognized for behavior recognition in this embodiment can be images collected separately for the same target object by image sensors in different working bands or different spatial positions within a preset small span time. As a result, there is no time constraint between the multiple images to be recognized, thus avoiding the influence of behavior changes in the video. Different working bands or different spatial positions can provide more image features for behavior recognition, thereby improving the accuracy of behavior recognition.

[0113] Based on the above embodiments, a second embodiment of the behavior recognition method of the present application is proposed. Figure 4, Figure 4 This is a flow chart of the second embodiment of the behavior recognition method of the present application.

[0114] In this embodiment, after step S200, the method further includes:

[0115] Step S300: output the recognition weight of each image to be recognized.

[0116] Step S400: Filter out unqualified points based on all recognition weights.

[0117] Specifically, in this embodiment, the behavior recognition network not only outputs the collective behavior recognition result obtained by the second behavior recognition module, but also outputs the recognition weight obtained by the recognition quality assessment module. That is, the recognition weight output by the softmas normalization layer in the recognition quality assessment module is output to the user. In this way, the image to be recognized with poor behavior recognition quality can be determined based on the size relationship of the recognition weight, and then the image sensor point corresponding to the image to be recognized with poor behavior recognition quality can be regarded as an unqualified point.

[0118] As an option of this embodiment, the points where the sensors of several corresponding images to be recognized that have obtained the minimum image recognition weight or the last ranked recognition weight can be regarded as unqualified points.

[0119] Alternatively, as another option of this embodiment, step S400 specifically includes:

[0120] Step S410: determining all identified target objects targeted by the image sensor within a preset working time, and the existing identification weight corresponding to each identified target object.

[0121] The preset working duration includes at least two preset small span durations.

[0122] Step S420: Filter out low-quality recognition weights from all existing recognition weights.

[0123] Among them, the low-quality recognition weight is less than the preset recognition weight.

[0124] Step S430: determine the proportion of all low-quality recognition weights in all existing recognition weights.

[0125] Step S440: If the proportion is less than the preset warning threshold, the point where the image sensor is located is determined as an unqualified point.

[0126] Specifically, in this option, within the preset working time, the behavior recognition device may continuously obtain images to be recognized obtained by imaging multiple different target objects with an image sensor. Taking the field of transportation as an example, within a designated road section, vehicles are constantly passing by, and a certain image sensor continuously images the passing vehicles, thereby obtaining images to be recognized of each driver. At this time, each target object is the driver of each vehicle. Since the preset working time includes at least two preset span durations, at least one target object is included in the relatively long preset duration. After being processed by the behavior recognition network, each target object is an identified target object. For each identified target object, the certain image sensor has a corresponding recognition weight.

[0127] In this way, when at least two target objects are included, the image sensor includes at least two existing recognition weights. At this time, the size of each existing recognition weight is determined respectively, and the existing recognition weight that is less than the preset recognition weight is determined as a low-quality recognition weight. If the preset recognition weight is 0.3, the existing recognition weight that is less than 0.3 can be regarded as a low-quality recognition weight.

[0128] Then, the proportion of the low-quality recognition weight in the at least two recognition weights can be calculated. If the proportion is less than the preset warning threshold, the point where the image sensor is located can be considered as an unqualified point.

[0129] Of course, it is understandable that the unqualified point may be a problem with the image sensor itself, or a problem with the image acquisition device to which the image sensor belongs, or a problem with the spatial position selection of the image sensor. At this time, maintenance personnel can conduct on-site inspections to analyze the specific reasons.

[0130] It is not difficult to see that compared with the inefficient full-scale detection of multiple front-end points in the related art, in this embodiment, the low-quality recognition weight ratio of a certain image sensor in a longer period of time relative to the preset small span time is reflected by the preset working time. The proportion of low-quality images obtained by a certain image sensor within the preset working time can thereby quickly and efficiently determine unqualified points from multiple front-end points.

[0131] Based on the above embodiments, a third embodiment of the behavior recognition method of the present application is proposed.

[0132] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0133] In this example, see Figure 5 , the behavior recognition network is trained as follows:

[0134] Step S10: Acquire at least two initial sample images.

[0135] The at least two initial sample images are respectively acquired by multiple image sensors at the same time with respect to the same sample object.

[0136] Step S20: Based on the initial sample image, generate an expanded sample image with similar image features.

[0137] Step S30: Obtain training samples based on the initial sample image and the expanded sample image.

[0138] Step S40: training the behavior recognition network using at least one training sample.

[0139] Specifically, in this embodiment, the initial sample image is acquired by multiple image sensors at the same time for the same sample object. After acquiring the initial sample image, the initial sample image can be subjected to image transformation processing, such as color transformation processing or brightness transformation processing to obtain a new image, that is, an expanded sample image. Alternatively, the initial sample image can also be input into an image generation model to obtain an expanded sample image with similar image features. For example, in one example, the image generation model is a Stable Diffusion model.

[0140] After the expanded sample images are obtained, training samples used for training can be constructed based on the initial sample images and the expanded sample images.

[0141] Then, a training sample set of the behavior recognition network can be constructed based on at least one training sample, and the behavior recognition network can be trained using the training sample set.

[0142] At this time, it is also necessary to manually mark each frame of the training sample. Manual marking can specifically include:

[0143] (1) Single-frame image calibration behavior category, that is, calibrating the second real behavior category of a single-frame image: For example, the clear & behavior category is X i , or is uncertain. Where i∈N, N is the total number of behavior categories;

[0144] (2) Calibration of the set behavior recognition results, that is, calibrating the first real behavior category of the training sample set, such as behavior category X i ;

[0145] (3) Identification weight calibration, that is, calibrating the true weight corresponding to each image: the quality ranking of each frame image.

[0146] Of course, it is understandable that the initial sample images within the multiple training samples of the training sample set should be imaged by at least two image sensors with different working bands and / or spatial positions for the same sample target object within a preset small span time length, that is, they can be imaged at the same moment, or adjacent moments, or within a preset small span time length.

[0147] In a specific implementation, step S40 specifically includes:

[0148] Step S41: input the training sample into the behavior recognition network to obtain the first predicted behavior category output by the behavior recognition network and at least two predicted recognition weights output by the recognition quality assessment module.

[0149] Step S42: construct a first loss function based on the first predicted behavior category and the first real behavior category of the training sample.

[0150] Step S43: construct a second loss function based on at least two predicted recognition weights and the true weight corresponding to each image in the training sample.

[0151] Step S44: training the behavior recognition network based on the first loss function and the second loss function.

[0152] Specifically, during the training process, the first loss function constructed is used to measure the collective behavior recognition loss of the second behavior recognition module, and the second loss function constructed is used to measure the recognition weight loss of the recognition quality evaluation module. In one example, the second loss function can be an MSEloss (Mean Squared Error) function.

[0153] The overall loss function constructed in this way can make the trained behavior recognition network not only perform well in collective behavior recognition, but also enable the trained behavior recognition network to accurately evaluate the recognition quality of a single frame image.

[0154] Further, step S44 specifically includes:

[0155] Step S441: Obtain a second predicted behavior category output by the first behavior recognition module for each image.

[0156] Step S442: Determine the recognition loss between the second predicted behavior category and the corresponding second real behavior category of the image.

[0157] Step S443: construct a third loss function based on the recognition losses of all images.

[0158] Step S444: training the behavior recognition network based on the first loss function, the second loss function and the third loss function.

[0159] That is, during the training process, the constructed loss function also includes a third loss function, which is used to measure the behavior recognition loss of a single-frame image of the first behavior recognition module, so that the obtained behavior recognition network can perform better in behavior recognition when facing a single-frame image, thereby improving the accuracy of the final collective behavior recognition.

[0160] The overall loss function of the final behavior recognition network can be expressed as:

[0161] loss_total=α*loss1+β*loss2+γ*loss3;

[0162] Among them, loss_total is the overall loss function, loss1 is the first loss function, loss2 is the second loss function, loss3 is the third loss function, α is the first weight of the first loss function, β is the second weight of the second loss function, γ is the third weight of the first loss function, and α+β+γ=1.

[0163] In order to enable those skilled in the art to better understand the scope of protection of the claims of this application, the following is an explanation of the technical solutions recorded in the claims of this application through specific implementation examples in specific application scenarios. It can be understood that the following examples are only used to explain this application, and are not used to limit the scope of protection of the claims of this application.

[0164] In one example, in the field of transportation, it is necessary to identify the behavior of the driver in the vehicle. At this time, the behavior recognition system includes multiple front-end devices and a back-end platform. All front-end devices are set up within a designated road section. The designated road section can be a 10m highway section. In the extension direction of the designated road section range, some of the front-end devices can be arranged at the same distance, such as being respectively set up on the left and right sides of the designated road section range, so as to shoot the passing vehicles from different directions at the same time to obtain the driver's image. Some of the front-end devices can be arranged in sequence along the extension direction, that is, the installation points are different, so as to shoot the passing vehicles at different times to obtain the driver's image. Of course, it is understandable that when the vehicle is traveling at the allowed driving speed of the designated road section range, the duration of the vehicle passing through the designated road section range is approximately equal to the preset small span duration. Due to the high speed of vehicles on the highway, the driver's behavior does not change much within the preset small span duration.

[0165] All front-end devices may have multiple types of image sensors, such as at least two of infrared image sensors, visible light image sensors, ultraviolet image sensors, radio wave image sensors, and microwave image sensors. Currently, one front-end device may have multiple types of image sensors. Image sensors with multiple working bands have different reflectivities for seat belts, human tops, mobile phones, cigarettes, cabin seats and interiors made of different materials, and can produce images with different contrasts.

[0166] All front-end devices send the collected multiple frames of driver images to the back-end platform through the network, so that the collected driver images can be identified on the back-end platform, such as identifying specific behaviors such as the driver not wearing a seat belt, making a phone call, smoking, etc. Multiple frames of driver images constitute a driver image set. It can be understood that the back-end platform can perform facial recognition and facial clustering on multiple frames of driver images, so as to eliminate images that do not belong to the same driver. In addition, the back-end platform can also verify the acquisition time, point information, and point topology information of each frame of the image to ensure that all images in the driver image set are collected by the front-end device mentioned above within a preset small span time. Obviously, there is no temporal relationship between the multiple frames of driver images in the driver image set.

[0167] The back-end platform inputs the driver image set into the behavior recognition network. The first behavior recognition module of the behavior recognition network outputs the predicted behavior recognition result, and sends the first vector feature of the predicted behavior recognition result to the second behavior recognition module. The recognition quality assessment module of the behavior recognition network outputs the recognition weight of each frame of the driver image, and sends the quality assessment vector of the recognition weight to the second behavior recognition module. The second behavior recognition module multiplies and accumulates all the first vector features and the corresponding quality assessment vectors to obtain the second vector features, and then normalizes the second vector features to obtain the collective behavior recognition result.

[0168] Therefore, the collective behavior recognition result in this embodiment fully considers the correlation of each frame of the driver image in the driver set, and mainly relies on the driver image that is more conducive to behavior recognition for behavior recognition. That is, the behavior recognition network can find that there are driver images with relatively clear and identifiable behaviors in the driver image set, and treat them in a targeted manner, distinguishing the impact of different images in the set on behavior recognition.

[0169] In addition, the collective behavior recognition results make full use of the imaging of graphic sensors in different working bands, which can improve the behavior recognition to the influence of lighting conditions, clothing, seat belts, cabin interior, etc. on image feature contrast, thereby improving the accuracy of behavior recognition.

[0170] In this example, the behavior recognition network can also output recognition weights in the form of quality scores so as to select the driver image with the best quality from the driver image collection and display it to the user.

[0171] In addition, outputting the recognition weight in the form of a quality score can also facilitate the back-end platform to count the imaging quality of the front-end equipment at each point over a longer period of time. For example, within one day, the quality score of the driver image collected by the front-end equipment of installation point A is always 0.1, which is significantly less than the preset recognition weight of 0.4. That is, the proportion of low-quality images is 100%. At this time, the installation point can be determined as an unqualified point, so that maintenance personnel can be arranged to conduct on-site inspections to determine whether the specific point setting is unreasonable or the front-end equipment has a fault (such as being blocked by foreign objects). Alternatively, within one day, the quality score of 800 driver images out of 1,000 driver images collected by the front-end equipment of installation point A is 0.1, which is significantly less than the preset recognition weight of 0.4. That is, the proportion of low-quality images is 80%, which is greater than the preset warning threshold of 60%. At this time, the installation point can also be determined as an unqualified point.

[0172] It is not difficult to see that in the field of transportation, there are many front-end devices, and it is difficult for maintenance personnel to conduct full-scale inspections, which is inefficient and prone to errors. However, by measuring the proportion of low-quality recognition weight images in the driver images output by the front-end devices, unqualified points can be accurately and efficiently determined, thereby reducing negative false alarms while ensuring positive recall.

[0173] Based on the same inventive concept, please refer to Figure 6 The present application also provides a behavior recognition device, the device comprising:

[0174] An image acquisition module, used to obtain a set of images to be identified based on images to be identified obtained by imaging the target object by at least two image sensors within a preset short span time, wherein at least two image sensors among all the image sensors have different working bands and / or spatial positions;

[0175] The network recognition module is used to input the set of images to be recognized into the trained behavior recognition network to obtain the collective behavior recognition result output by the behavior recognition network; wherein the behavior recognition network includes a feature extraction module, a first behavior recognition module, a recognition quality assessment module and a second behavior recognition module; the feature extraction module is used to extract features from each image to be recognized to obtain image features; the first behavior recognition module is used to perform behavior recognition on each image to be recognized based on the image features to obtain the predicted behavior recognition result of each image to be recognized; the recognition quality assessment module is used to perform recognition quality assessment on each image to be recognized to obtain the recognition weight of each image to be recognized; the second behavior recognition module is used to obtain the collective behavior recognition result based on the predicted behavior recognition result and the corresponding recognition weight.

[0176] In a possible embodiment of the present application, the image acquisition module specifically includes:

[0177] An image acquisition unit, used to acquire multiple frames of real-time images collected by an image sensor at different times within a preset short span time;

[0178] A clustering unit, used for performing feature recognition and feature clustering on multiple frames of real-time images to obtain feature clustering results;

[0179] An image screening unit, used for screening out the image to be identified including the target object from the multiple frames of real-time images based on the feature clustering result;

[0180] The set construction unit is used to obtain a set of images to be identified based on the images to be identified from at least two image sensors.

[0181] In a possible embodiment of the present application, the behavior recognition device further includes:

[0182] A weight output module, used to output the recognition weight of each image to be recognized;

[0183] The point screening module is used to screen out unqualified points based on all recognition weights.

[0184] In a possible embodiment of the present application, the point screening module specifically includes:

[0185] A data statistics unit, used to determine all identified target objects targeted by the image sensor within a preset working time, and the existing recognition weights corresponding to each identified target object; the preset working time includes at least two preset small span time lengths;

[0186] A weight screening unit, used to screen out low-quality recognition weights from all existing recognition weights; wherein the low-quality recognition weight is less than a preset recognition weight;

[0187] A proportion calculation unit, used to determine the proportion of all low-quality recognition weights in all existing recognition weights;

[0188] The point screening unit is used to determine the point where the image sensor is located as an unqualified point if the proportion is less than a preset warning threshold.

[0189] In a possible embodiment of the present application, the behavior recognition device further includes:

[0190] A sample image acquisition module, used to acquire at least two initial sample images; wherein the at least two initial sample images are acquired by multiple image sensors at the same time and for the same sample object;

[0191] An image expansion module, used for generating an expanded sample image with similar image features based on an initial sample image;

[0192] A sample construction module, used to obtain training samples based on the initial sample image and the expanded sample image;

[0193] The training module is used to train the behavior recognition network using at least one training sample.

[0194] In a possible embodiment of the present application, the training module is specifically used to: input the training sample into the behavior recognition network to obtain the first predicted behavior category output by the behavior recognition network and at least two predicted recognition weights output by the recognition quality assessment module; construct a first loss function based on the first predicted behavior category and the first real behavior category of the training sample; construct a second loss function based on the at least two predicted recognition weights and the real weights corresponding to each image in the training sample; and train the behavior recognition network based on the first loss function and the second loss function.

[0195] In a possible embodiment of the present application, the training module is further specifically used to obtain the second predicted behavior category output by the first behavior recognition module for each image; determine the recognition loss between the second predicted behavior category and the second real behavior category of the corresponding image; construct a third loss function based on the recognition loss of all images; and train the behavior recognition network based on the first loss function, the second loss function and the third loss function.

[0196] It should be noted that the various implementations of the behavior recognition device in this embodiment and the technical effects achieved can refer to the various implementations of the behavior recognition method in the aforementioned embodiments, and will not be repeated here.

[0197] In addition, an embodiment of the present application also proposes a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the behavior recognition method as described above are implemented. Therefore, it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of the present application. As an example, program instructions can be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0198] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0199] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0200] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method of each embodiment of the present application.

[0201] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A behavior recognition method, characterized in that: The method comprises: A set of images to be identified is obtained based on images to be identified obtained by imaging the target object respectively by at least two image sensors within a preset short span time; wherein at least two of all the image sensors have different working bands and / or spatial positions; The set of images to be recognized is input into a trained behavior recognition network to obtain a collective behavior recognition result output by the behavior recognition network; wherein the behavior recognition network includes a feature extraction module, a first behavior recognition module, a recognition quality assessment module and a second behavior recognition module; the feature extraction module is used to extract features from each of the images to be recognized to obtain image features; the first behavior recognition module is used to perform behavior recognition on each of the images to be recognized based on the image features to obtain a predicted behavior recognition result of each of the images to be recognized; the recognition quality assessment module is used to perform recognition quality assessment on each of the images to be recognized to obtain a recognition weight of each of the images to be recognized; the second behavior recognition module is used to obtain the collective behavior recognition result based on the predicted behavior recognition result and the corresponding recognition weight.

2. The behavior recognition method according to claim 1, characterized in that: The step of obtaining a set of images to be identified based on the images to be identified obtained by imaging the target object respectively by at least two image sensors comprises: Acquire multiple frames of real-time images collected by the image sensor at different times within the preset short span time; Performing feature recognition and feature clustering on multiple frames of the real-time image to obtain feature clustering results; Based on the feature clustering result, selecting the image to be identified including the target object from the multiple frames of real-time images; The set of images to be recognized is obtained based on the images to be recognized from at least two of the image sensors.

3. The behavior recognition method according to claim 1, characterized in that: After inputting the set of images to be recognized into a trained behavior recognition network and obtaining a set behavior recognition result output by the behavior recognition network, the method further includes: Outputting the recognition weight of each of the images to be recognized; Based on all the identification weights, unqualified points are screened out.

4. The behavior recognition method according to claim 3, characterized in that: The step of screening out unqualified points based on all the identification weights includes: Determine all identified target objects targeted by the image sensor within a preset working time, and the existing recognition weights corresponding to each of the identified target objects; the preset working time includes at least two of the preset small span time lengths; Filtering out a low-quality recognition weight from all the existing recognition weights; wherein the low-quality recognition weight is less than a preset recognition weight; Determine the proportion of all the low-quality recognition weights in all the existing recognition weights; If the proportion is less than a preset warning threshold, the point where the image sensor is located is determined as the unqualified point.

5. The behavior recognition method according to claim 1, characterized in that: Before obtaining the set of images to be identified based on the images to be identified obtained by imaging the target object by at least two image sensors within a preset short span time, the method further includes: Acquire at least two initial sample images; wherein at least two of the initial sample images are acquired by a plurality of the image sensors at the same time and for the same sample object; Based on the initial sample image, generating an expanded sample image having similar image features; Based on the initial sample image and the expanded sample image, obtaining a training sample; The behavior recognition network is trained using at least one of the training samples.

6. The behavior recognition method according to claim 5, characterized in that: The step of training the behavior recognition network using at least one of the training samples includes: Inputting the training sample into the behavior recognition network, obtaining a first predicted behavior category output by the behavior recognition network and at least two predicted recognition weights output by the recognition quality assessment module; constructing a first loss function based on the first predicted behavior category and the first real behavior category of the training sample; constructing a second loss function based on at least two of the predicted recognition weights and a true weight corresponding to each image in the training sample; The behavior recognition network is trained based on the first loss function and the second loss function.

7. The behavior recognition method according to claim 6, characterized in that: The training of the behavior recognition network based on the first loss function and the second loss function includes: Obtaining a second predicted behavior category output by the first behavior recognition module for each of the images; determining a recognition loss between the second predicted behavior category and a corresponding second true behavior category of the image; Constructing a third loss function based on the recognition losses of all the images; The behavior recognition network is trained based on the first loss function, the second loss function and the third loss function.

8. A behavior recognition device, characterized in that: The device comprises: An image acquisition module, configured to obtain a set of images to be identified based on images to be identified obtained by imaging a target object by at least two image sensors within a preset short span time, wherein at least two of the image sensors have different working bands and / or spatial positions; A network recognition module is used to input the set of images to be recognized into a trained behavior recognition network to obtain a collective behavior recognition result output by the behavior recognition network; wherein the behavior recognition network includes a feature extraction module, a first behavior recognition module, a recognition quality assessment module and a second behavior recognition module; the feature extraction module is used to extract features from each of the images to be recognized to obtain image features; the first behavior recognition module is used to perform behavior recognition on each of the images to be recognized based on the image features to obtain a predicted behavior recognition result of each of the images to be recognized; the recognition quality assessment module is used to perform recognition quality assessment on each of the images to be recognized to obtain a recognition weight of each of the images to be recognized; the second behavior recognition module is used to obtain the collective behavior recognition result based on the predicted behavior recognition result and the corresponding recognition weight.

9. A behavior recognition device, characterized in that: include: A processor, a memory, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the behavior recognition method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the behavior recognition method according to any one of claims 1 to 7 is implemented.

Images