Behavior recognition method, device, medium and electronic equipment

Abstract

Disclosed are a behavior recognition method, device, medium and equipment, wherein the method comprises: obtaining an image block sequence containing a target part of a target object from a plurality of video frames acquired by a camera device disposed in a mobile device; performing feature vector extraction processing based on a first preset behavior on the image block sequence to obtain a first feature vector corresponding to the image block sequence; determining the behavior type to which the behavior of the target object belongs based on the first feature vector; if the behavior type to which the behavior of the target object belongs conforms to the first behavior type, determining the behavior of the target object based on the result of behavior recognition processing of the image block sequence for a second preset behavior. The technical solution provided by the present disclosure can realize cascade recognition of the behavior of the target object, thereby facilitating improving the accuracy of behavior recognition, and further facilitating improving the driving safety of the mobile device.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and in particular to a behavior recognition method, a behavior recognition device, a storage medium, and an electronic device. Background Art

[0002] In the field of driving, since certain behaviors of drivers and passengers of mobile devices may affect the safe driving of the mobile device, it is very necessary to identify the behaviors of target objects in the mobile device and take timely measures when the behaviors of the identified target objects are not conducive to the safe driving of the mobile device, so as to minimize the impact of the behaviors of the target objects on the safe driving of the mobile device. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a behavior recognition method, device, storage medium and electronic device.

[0004] According to one aspect of an embodiment of the present disclosure, a behavior recognition method is provided, comprising: obtaining an image block sequence including a target part of a target object from a plurality of video frames acquired by a camera device disposed in a mobile device; performing feature vector extraction processing based on a first preset behavior on the image block sequence to obtain a first feature vector corresponding to the image block sequence; determining a behavior type to which the behavior of the target object belongs based on the first feature vector; if the behavior type to which the behavior of the target object belongs conforms to the first behavior type, determining the behavior of the target object based on a result of behavior recognition processing of the image block sequence for a second preset behavior.

[0005] According to another aspect of an embodiment of the present disclosure, a behavior recognition device is provided, including: a sequence acquisition module, for acquiring an image block sequence including a target part of a target object from a plurality of video frames acquired by a camera device disposed in a mobile device; a feature vector acquisition module, for performing feature vector extraction processing based on a first preset behavior on the image block sequence acquired by the sequence acquisition module, to obtain a first feature vector corresponding to the image block sequence; a behavior type determination module, for determining a behavior type to which the behavior of the target object belongs based on the first feature vector obtained by the feature vector acquisition module; and a cascade recognition module, for determining the behavior of the target object based on a result of behavior recognition processing of the image block sequence for a second preset behavior if the behavior type to which the behavior of the target object determined by the behavior type determination module belongs conforms to the first behavior type.

[0006] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to implement the above method.

[0007] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the above method.

[0008] Based on a behavior recognition method and device provided by the above-mentioned embodiments of the present disclosure, by extracting image blocks of target parts of target objects such as drivers or passengers in the mobile device from video frames captured by a camera device set in the mobile device, and obtaining a first feature vector from the image block sequence, the present disclosure can obtain the behavior type to which the behavior of the target object belongs based on the first feature vector. Since when the behavior type to which the behavior of the target object belongs conforms to the first behavior type (such as a suspected behavior type), the present disclosure can determine the behavior of the target object in combination with the result of the behavior recognition processing corresponding to the second preset behavior performed on the image block sequence. Therefore, the present disclosure realizes cascade recognition of the behavior of the target object, which is conducive to avoiding the influence of the threshold used for behavior recognition being set too high or too low on the accuracy of behavior recognition. It can be seen that the technical solution provided by the present disclosure is conducive to improving the accuracy of behavior recognition, and by taking corresponding voice prompts, light prompts or vibration prompts and other measures for the identified behavior of the target object (such as dangerous driving behavior or harmful driving behavior that affects the driver's driving, etc.), it is conducive to improving the driving safety of the mobile device.

[0009] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] By describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present disclosure will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 is a schematic diagram of an embodiment of a DMS structure in a vehicle of the present disclosure;

[0012] Figure 2 A flowchart of an embodiment of the behavior recognition method disclosed in the present invention;

[0013] Figure 3A and Figure 3BA schematic diagram of an embodiment of the first preset behavior and the second preset behavior of the present disclosure;

[0014] Figure 4 It is a structural schematic diagram of an embodiment of a common block disclosed in the present invention;

[0015] Figure 5 It is a structural schematic diagram of an embodiment of a downsampling block disclosed in the present invention;

[0016] Figure 6 A flowchart of an embodiment of the present disclosure for obtaining an image block sequence including a target part of a target object;

[0017] Figure 7 A schematic diagram of an embodiment of obtaining an image block sequence including a target part of a target object disclosed in the present invention;

[0018] Figure 8 A flowchart of an embodiment of determining the behavior type to which the behavior of a target object belongs according to the present disclosure;

[0019] Fig. 9 A flowchart of an embodiment of determining the behavior of a target object according to the present disclosure;

[0020] Fig.10 It is a structural schematic diagram of an embodiment of the behavior recognition device disclosed in the present invention;

[0021] Fig.11 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The exemplary embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0023] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0024] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0025] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0026] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0027] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0028] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0029] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0030] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0031] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0032] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0033] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0034] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks can be performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0035] SUMMARY OF THE DISCLOSURE

[0036] In the process of implementing the present disclosure, the inventors found that the DMS (Driver Monitor System) in the mobile device often implements behavior recognition based on a preset threshold. For example, after the DMS processes the video frames collected by the camera device, it uses a preset threshold for judging whether the current behavior is a dangerous driving behavior to measure the processing result, thereby determining whether the driver's current behavior is a dangerous driving behavior.

[0037] In actual applications, similar behaviors often exist. For example, smoking and using a straw are similar, and smoking is a dangerous driving behavior, while using a straw is not. If the threshold for determining whether the current behavior is a dangerous driving behavior is set too high, dangerous driving behaviors may be missed, and if the threshold for determining whether the current behavior is a dangerous driving behavior is set too low, dangerous driving behaviors may be falsely detected.

[0038] Due to the diversity of behaviors of different target objects, it is often difficult to accurately set the threshold for determining whether the current behavior is a dangerous driving behavior. However, if cascaded behavior recognition can be performed for the target objects in the video frame, it will help improve the accuracy of behavior recognition.

[0039] Exemplary Overview

[0040] The technical solution of behavior recognition disclosed in the present invention can be applied to a variety of scenarios. For example, the technical solution of behavior recognition provided by the present invention can be applied to DMS applications. Figure 1 shown.

[0041] Combine the following Figure 1 , the application of the behavior recognition technology disclosed in the present invention is explained.

[0042] Figure 1In the present invention, at least one camera device 100 and a DMS 101 are arranged in a vehicle, and the video frames acquired by the camera device 100 can be provided to the DMS 101 in real time. When the driver is in the driving position of the vehicle, the driver's face should be within the field of view of the camera device 100, that is, the video acquired by the camera device 100 usually includes the driver's face (such as the front face, etc.).

[0043] Combine the following Figure 1 , the application of the fatigue state detection technology disclosed in the present invention is explained.

[0044] Figure 1 In the embodiment, a vehicle is used as an example for illustrative description. A vehicle is equipped with at least one camera 100 and a DMS 101. The video frames acquired by the camera 100 can be provided to the DMS 101 in real time. When the driver is in the driving position of the vehicle, the driver's face should be within the field of view of the camera 100, that is, the video acquired by the camera 100 usually includes the driver's face (such as the front face, etc.) area.

[0045] DMS101 at least includes a receiving module 1011, a behavior recognition device 1012 and a behavior control module 1013. After receiving the video frame transmitted in real time by the camera device 100, the receiving module 1011 triggers the behavior recognition device 1012 to perform dangerous driving behavior recognition processing on the received video frame. The behavior recognition device 1012 determines the behavior of the target object in multiple video frames in each time window (such as 2 seconds, etc.). If the behavior of the target object determined by the behavior recognition device 1012 is a dangerous driving behavior, the behavior control module 1013 can prompt the driver through voice prompts, light prompts or vibration prompts to ensure the safe driving of the vehicle.

[0046] During the process of the behavior recognition device 1012 performing the dangerous driving behavior recognition processing, if it is determined that the driver's current behavior is other behaviors similar to the dangerous driving behavior, the behavior recognition device 1012 can perform cascade behavior recognition processing on the driver's current behavior to determine whether the driver's current behavior is other behaviors similar to the dangerous driving behavior. If it is other behaviors similar to the dangerous driving behavior, the behavior recognition device 1012 can avoid the false alarm of the dangerous driving behavior. If it is not other behaviors similar to the dangerous driving behavior, the behavior recognition device 1012 can consider that the driver's current behavior is a dangerous driving behavior, thereby avoiding the underreporting of the dangerous driving behavior.

[0047] In addition, the technical solution for behavior recognition provided by the present disclosure can also be applied to IMS (In-cabin Monitoring System) applications. For example, the present disclosure can use the IMS in the vehicle to identify the behavior of people in the co-pilot seat or the back seat of the vehicle, so as to identify the behavior of people in the co-pilot seat or the back seat of the vehicle that is not conducive to the safe driving of the vehicle. The technical solution provided by the present disclosure is applicable to the specific implementation process in IMS applications, which is basically the same as the above-mentioned specific implementation process applicable to DMS applications, and will not be described in detail here.

[0048] Exemplary Methods

[0049] Figure 2 FIG. 1 is a flow chart of an embodiment of the behavior recognition method disclosed in the present invention. Figure 2 The method shown mainly includes: S200, S201, S202 and S203. Each step is described below.

[0050] S200: Acquire a sequence of image blocks including a target part of a target object from a plurality of video frames acquired by a camera device disposed in a mobile device.

[0051] The mobile device in the present disclosure may refer to a vehicle capable of carrying people, such as a private car and a public transportation vehicle, etc. The target object in the present disclosure may generally be the driver of the mobile device, and of course, the target object may also be other people in the mobile device, such as a person in the co-pilot seat or a passenger in a public transportation vehicle, etc.

[0052] In the case where the target object is the driver of the mobile device, the camera device in the present disclosure can be installed at the steering column of the mobile device or the A-pillar on the driver's side. The installation position of the camera device should ensure that the face (such as the front face) of the driver at the driving position is within the field of view of the camera device. In the case where the target object is a passenger in the mobile device, the installation position of the camera device can be set according to actual needs. For example, the camera device can be installed at the A-pillar on the co-pilot side of the mobile device to ensure that the face (such as the front face) of the passenger at the co-pilot position is within the field of view of the camera device. For another example, the camera device can be installed at the B-pillar or the back of the front seat of the mobile device to ensure that the face (such as the front face) of the passenger in the back row is within the field of view of the camera device. In addition, multiple cameras can be installed in the mobile device to realize real-time shooting of the driver and at least one passenger, so that the behavior recognition of the driver and at least one passenger can be realized based on the video frames acquired by each camera device.

[0053] The camera device in the present disclosure may be a camera device based on RGB (Red Green Blue) or a camera device based on IR (Infrared Radiation), etc. In addition, the camera device may be a monocular camera device or a binocular camera device, etc.

[0054] The multiple video frames in the present disclosure may be all video frames in a predetermined time window (such as n seconds, where n is a positive number); or may be some video frames from all video frames in a predetermined time window; in one example, the multiple video frames in the present disclosure may be multiple video frames containing the face of a target object selected from all video frames in a predetermined time window.

[0055] The target part of the target object in the present disclosure is usually related to the behavior that needs to be identified, that is, the target part of the target object in the present disclosure is usually the part involved in the behavior that needs to be identified, and the target part of the target object in the present disclosure may include: one part or multiple different parts on the body of the target object.

[0056] For example, if the behavior to be identified includes smoking behavior, the target part of the target object may include: the lower part of the face, for example, the lower part of the face including the nose and the mouth.

[0057] For another example, if the behavior to be identified includes smoking behavior, the target parts of the target object may include: the lower part of the face and the hand.

[0058] As another example, if the behavior to be recognized includes making a phone call, the target part of the target object may include: the left / right side of the face, for example, the left / right side of the face including the left / right cheek.

[0059] As another example, if the behavior to be recognized includes making a phone call, the target parts of the target object may include: the left / right side of the face and the hand, etc.

[0060] The present disclosure does not limit the specific expression form of the target part of the target object contained in each image block in the image block sequence.

[0061] The image block sequence containing the target part of the target object obtained by the present invention can be: an image block sequence consisting of image blocks of the same size containing the target part of the target object respectively captured from each video frame and arranged in the order of the acquisition time of the video frames.

[0062] In addition, the image block sequence containing the target part of the target object obtained from multiple video frames in the present disclosure may be one image block sequence or multiple image block sequences, and different image block sequences usually correspond to different behaviors that need to be identified. In one example, if the behaviors that need to be identified include: smoking behavior and phone call behavior, the present disclosure may obtain a first image block sequence for the smoking behavior, and obtain a second image block sequence for the phone call behavior. For example, the first image block sequence may be an image block sequence formed by multiple image blocks that respectively contain the lower part of the face and the hand part. For another example, the second image block sequence may be an image block sequence formed by multiple image blocks that respectively contain the left / right side of the face and the hand part.

[0063] The present disclosure can obtain image blocks containing target parts of target objects in each video frame by performing image recognition and other processing on multiple video frames respectively, thereby obtaining an image block sequence based on the target parts. For example, for any video frame, the present disclosure can obtain at least one ROI (Region Of Interest) in the video frame through a first neural network for image recognition, and perform a cropping process on the video frame according to the at least one ROI (such as forming a cropping frame based on multiple ROIs, and cropping the video frame based on the cropping frame), thereby obtaining an image block of the video frame. After the cropping process is performed on each video frame, an image block sequence based on the target part is obtained.

[0064] In one example, the first neural network may be a neural network based on region detection, such as Faster RCNN (Regions with Convolutional Neural Networks), etc. The computational processing of the neural network in the present disclosure may be implemented by a data processing unit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a BPU (Brain Processing Unit).

[0065] S201 . Perform feature vector extraction processing based on a first preset behavior on an image block sequence to obtain a first feature vector corresponding to the image block sequence.

[0066] The first preset behavior in the present disclosure may include multiple behaviors that need to be identified. For example, the first preset behavior may include: at least one dangerous behavior or prohibited behavior or prescribed behavior, at least one non-dangerous behavior or non-prohibited behavior or non-prescribed behavior, at least one suspected dangerous behavior or suspected prohibited behavior or suspected prescribed behavior, etc. At least one behavior among the multiple behaviors included in the first preset behavior in the present disclosure belongs to the first behavior type, and the remaining behaviors may belong to the second behavior type.

[0067] The feature vector extraction process based on the first preset behavior in the present disclosure is a feature vector extraction method, and the feature vector extraction method is a feature vector extraction method set for the purpose of identifying the first preset behavior. For example, the present disclosure can use a neural network to implement feature vector extraction processing. The present disclosure can use training samples with first preset behavior annotation information to train the neural network. After the neural network is successfully trained, the network parameters of the neural network are obtained. The feature vector extraction process performed by the neural network processing unit using the network parameters is the feature vector extraction process based on the first preset behavior.

[0068] An example of a process of using a neural network to extract a feature vector based on a first preset behavior for an image block sequence in the present disclosure may be: each image block in the image block sequence is respectively processed by a second neural network for extracting features, and a feature vector (such as a one-dimensional array formed by multiple vector elements for describing the characteristics of a target part) is obtained for each image block in the image block sequence based on the calculation result of the second neural network, and the feature vectors of all image blocks are the first feature vectors corresponding to the image block sequence.

[0069] The second neural network in the present disclosure may be a convolutional neural network, etc., and the present disclosure may use one second neural network to obtain the feature vectors of each image block, or may use multiple second neural networks in parallel to obtain the feature vectors of each image block. Multiple second neural networks usually have the same network structure and the same network parameters. It should be noted that the operations of the first neural network and the second neural network can be implemented by the same neural network processing unit, such as by the same BPU or the same GPU. Similarly, the neural networks based on different operation processing involved in the embodiments of the present disclosure can all be implemented by the same neural network processing unit, and will not be described one by one below.

[0070] S202: Determine the behavior type of the target object's behavior according to the first feature vector.

[0071] The behavior type in the present disclosure may refer to a behavior type formed by classifying behaviors. The present disclosure sets at least two behavior types for the behaviors included in the first preset behavior, one of which is the first behavior type.

[0072] In an example, the first behavior type may refer to a type that cannot accurately determine whether the behavior of the target object is a specific behavior (such as a dangerous behavior, etc.). The first behavior type may be called a suspected specific behavior type (such as a suspected dangerous behavior type, etc.).

[0073] In one example, the suspected specific behavior type may specifically be a suspected smoking behavior type or a suspected phone call behavior type, etc. In the case where there are multiple specific behaviors, the first behavior type in the present disclosure may include multiple suspected specific behaviors.

[0074] The present disclosure can obtain a behavior classification result by performing behavior classification processing on the first feature vector of the image block sequence, and judge the behavior classification result by using the conditions pre-set for the suspected specific behaviors included in the first behavior type and the conditions pre-set for the specific behaviors included in the second behavior type, so as to determine the behavior type to which the behavior of the target object belongs.

[0075] In one example, the present disclosure can utilize a third neural network for behavior classification to perform behavior classification processing on the first feature vector, and determine the behavior type to which the behavior of the target object belongs based on the operation results of the third neural network, the conditions pre-set for the suspected specific behaviors included in the first behavior type, and the conditions pre-set for the specific behaviors included in the second behavior type.

[0076] S203: If the behavior type to which the behavior of the target object belongs conforms to the first behavior type, determine the behavior of the target object according to a result of performing behavior recognition processing of a second preset behavior on the image block sequence.

[0077] The process of performing behavior recognition processing of the second preset behavior on the image block sequence in the present disclosure can be executed when it is determined that the behavior type to which the behavior of the target object belongs is consistent with the first behavior type, that is, determining that the behavior type to which the behavior of the target object belongs is consistent with the first behavior type can be a trigger condition for performing behavior recognition processing of the second preset behavior on the image block sequence.

[0078] The process of performing the behavior recognition processing of the second preset behavior on the image block sequence in the present disclosure can also be performed in parallel with S202, that is, regardless of whether the behavior type to which the behavior of the target object belongs conforms to the first behavior type, the present disclosure can perform the behavior recognition processing of the second preset behavior on the image block sequence while determining the behavior type to which the behavior of the target object belongs according to the first feature vector. In other words, determining that the behavior type to which the behavior of the target object belongs conforms to the first behavior type is no longer a trigger condition for performing the behavior recognition processing of the second preset behavior on the image block sequence.

[0079] The second preset behavior in the present disclosure may refer to a behavior similar to a specific behavior included in the first preset behavior. In one example, the first preset behavior includes a smoking behavior (such as Figure 3A In the case of a situation as shown in FIG. 1 , the second preset behavior may include: a behavior similar to a smoking behavior, for example, a straw-using behavior (such as Figure 3B In another example, when the first preset behavior includes a calling behavior, the second preset behavior may include: a behavior similar to the calling behavior, for example, using an instant message behavior. The present disclosure does not limit the specific manifestations of the first preset behavior and the second preset behavior.

[0080] The process of the present disclosure performing the behavior recognition processing of the second preset behavior on the image block sequence can be basically the same as the process of performing the behavior recognition processing of the first preset behavior on the image block sequence. For example, the present disclosure can perform the feature vector extraction processing based on the second preset behavior on the image block sequence, obtain the second feature vector corresponding to the image block sequence, and judge whether the behavior of the target object is the second preset behavior according to the second feature vector. Afterwards, the present disclosure can finally determine the behavior of the target object based on the result of this judgment.

[0081] The present disclosure extracts image blocks of target parts of target objects such as drivers or passengers in the mobile device from video frames captured by a camera device disposed in the mobile device, and obtains a first feature vector from an image block sequence, so that the present disclosure can obtain the behavior type to which the behavior of the target object belongs based on the first feature vector. Since, when the behavior type to which the behavior of the target object belongs conforms to the first behavior type (such as a suspected behavior type, etc.), the present disclosure can determine the behavior of the target object in combination with the result of the behavior recognition processing corresponding to the second preset behavior performed on the image block sequence. Therefore, the present disclosure realizes cascade recognition of the behavior of the target object, which is conducive to avoiding the influence of the threshold used for behavior recognition being set too high or too low on the accuracy of behavior recognition. It can be seen that the technical solution provided by the present disclosure is conducive to improving the accuracy of behavior recognition, and by taking corresponding voice prompts, light prompts, vibration prompts and other measures for the identified behavior of the target object (such as dangerous driving behavior or harmful driving behavior that affects the driver's driving, etc.), it is conducive to improving the driving safety of the mobile device.

[0082] Optionally, the second neural network in the present disclosure may include VargNet (variable group network). In one example, VargNet may be a neural network formed by connecting at least two blocks, and the two blocks may be a normal block and a down sampling block.

[0083] Optionally, a common block may include: at least two first units connected in series, and the first unit includes: a variable group convolution (Variable Group Conv, also called a variable group convolution layer) layer and a first convolution layer (such as a 1×1 point-by-point convolution layer), and the channel dimension of the input data of the first convolution layer is twice the channel dimension of the output data. An example of a common block including two first units connected in series is as follows Figure 4 As shown, Figure 4 400 in represents a variable group convolution layer (whose output channel dimension is twice the input channel dimension), and 401 represents the first convolution layer. The output of the last unit in a common block can be fused with the input of the common block (e.g. Figure 4 402), thereby forming the output of a common block.

[0084] Optionally, the downsampling block may include: at least one first unit and at least one second unit, and the first unit may include: a variable group of convolutional layers and a first convolutional layer, and the channel dimension of the input data of the first convolutional layer is twice the channel dimension of the output data. The second unit may include: a variable group of convolutional layers and a second convolutional layer (such as a point-by-point convolutional layer). The channel dimension of the input data of the second convolutional layer is the same as the channel dimension of the output data. All the first units and the second units are combined in series and parallel to form a downsampling block. An example of a downsampling block is as follows Figure 5 shown. Figure 5 500 in FIG. 5 represents a variable group convolution layer, 501 represents a first convolution layer, and 502 represents a second convolution layer.

[0085] In an optional example, the present disclosure obtains an example of an image block sequence containing a target part of a target object, such as Figure 6 and Figure 7 shown.

[0086] Figure 6 In S600, for any one of a plurality of video frames acquired from a camera device disposed in a mobile device, an image region containing at least one target part of a target object in the video frame is acquired to obtain at least one image region.

[0087] Optionally, for any video frame (such as Figure 7 For example, the present invention can obtain at least one image region of the video frame by identifying key points. For example, the present invention can obtain the hand image region (such as the hand region) of the target object in the video frame by identifying the skeleton key points. Figure 7 The face image region (e.g., ROI) of the target object in the video frame can be obtained by recognizing key points of the face. Figure 7 region 702 in, i.e., another ROI, Figure 7 Only a few facial key points are schematically shown), thereby obtaining two image regions, namely two ROIs.

[0088] In one example, the present disclosure may successively perform operations on each video frame through a first neural network, and obtain the ROI in each video frame according to the operation results of the first neural network.

[0089] In another example, the present disclosure can perform parallel operations on each video frame through multiple first neural network processing, and obtain the ROI in each video frame in parallel according to the operation results of the multiple first neural networks.

[0090] S601: Determine a target area of ​​the video frame according to the at least one image area obtained above.

[0091] Optionally, the present disclosure may obtain a target area of ​​a video frame according to coordinate values ​​of each vertex of each image area in the video frame.

[0092] In one example, if n1 (n1 is an integer greater than 1) image areas are obtained from a video frame, and the n1 image areas have an intersection, the present disclosure can take the maximum coordinate value and the minimum coordinate value of all vertices of the n1 image areas in the height direction, and take the minimum coordinate value / maximum coordinate value of all vertices of the n1 image areas in the length direction and the maximum coordinate value / minimum coordinate value in the intersection area, so as to form the coordinate values ​​of four points, and the area formed by the coordinate values ​​of the four points is the target area of ​​the video frame, such as Figure 7 The target area 703 in.

[0093] In another example, if only one image region is obtained from a video frame, the present disclosure may use the image region as the target region of the video frame.

[0094] In another example, if n1 image areas are obtained from a video frame and there is no intersection among the n1 image areas, the present disclosure can take the maximum coordinate value and the minimum coordinate value of all vertices of the n1 image areas in the height direction, and take the maximum coordinate value and the minimum coordinate value of all vertices of the n1 image areas in the length direction, thereby forming the coordinate values ​​of four points, and the area formed by the coordinate values ​​of the four points is the target area of ​​the video frame.

[0095] The present disclosure does not limit the specific implementation method of obtaining the target area of ​​the video frame.

[0096] S602: Obtain an image block of the video frame including a target part of the target object according to the target area of ​​the video frame.

[0097] Optionally, for a video frame, the present disclosure can directly cut out the target area of ​​the video frame from the video frame, thereby obtaining an image block of the video frame containing the target part of the target object. The present disclosure can also perform an expansion process on the target area of ​​the video frame, for example, the length and width of the target area of ​​the video frame are each expanded by n2 (n2 is an integer greater than 1, such as n2 is an even number such as 10) pixels to obtain the expanded target area, and cut out the expanded target area from the video frame, thereby obtaining an image block of the video frame containing the target part of the target object.

[0098] The present invention forms a target area of ​​a video frame by utilizing an image area containing at least one target part of a target object in a video frame, and finally forms an image block containing the target part of the target object. It can conveniently generate an image block sequence for complex actions involving multiple parts, thereby helping to improve the ease of use of behavior recognition.

[0099] In an optional example, the present disclosure determines an example of a behavior type to which the behavior of the target object belongs based on the first feature vector. Figure 8 shown.

[0100] Figure 8 In S800, for any image block in the image block sequence, according to a first feature vector of the image block, the confidences of the behaviors of the target object being a plurality of first preset behaviors are determined to obtain a plurality of first confidences.

[0101] Optionally, the present disclosure may respectively perform calculations on the first feature vector of each image block through a third neural network for behavior classification, and obtain multiple confidences corresponding to each image block based on the calculation results of the third neural network. For example, the first preset behavior includes n3 (n3 is an integer greater than 1) behaviors (i.e., the number of the first preset behaviors is n3, for example, smoking behavior, suspected smoking behavior, and non-smoking behavior, these 3 behaviors), and the present disclosure may use the calculation results of the third neural network to obtain n3 confidences corresponding to each image block, and the n3 confidences corresponding to one image block are a group of confidences. The third neural network in the present disclosure is usually combined with the second neural network and trained together.

[0102] Optionally, at least one of the multiple first preset behaviors in the present disclosure belongs to the first behavior type, and at least one of the multiple first preset behaviors belongs to the second behavior type. In one example, when the number of the multiple first preset behaviors is 3, and the three first preset behaviors are respectively: smoking behavior, suspected smoking behavior, and non-smoking behavior, the suspected smoking behavior belongs to the first behavior type, and the smoking behavior and non-smoking behavior belong to the second behavior type. In another example, when the number of the multiple first preset behaviors is 3, and the three first preset behaviors are respectively: calling behavior, suspected calling behavior, and non-calling behavior, the suspected calling behavior belongs to the first behavior type, and the calling behavior and non-calling behavior belong to the second behavior type.

[0103] S801: Determine a behavior type to which a behavior of a target object belongs according to a plurality of first confidences corresponding to each image block in an image block sequence and a first threshold corresponding to a first preset behavior.

[0104] Optionally, the present disclosure may calculate and process multiple confidences corresponding to each image block, thereby obtaining multiple confidences corresponding to the image block sequence (such as n3 confidences). The present disclosure may determine whether the magnitude relationship between the multiple confidences corresponding to the image block sequence and the first threshold satisfies a predetermined condition, and determine the behavior of the target object in the multiple video frames according to the determination result, thereby determining the behavior type to which the behavior of the target object belongs.

[0105] In an example, the present disclosure may calculate the mean of the first confidences (such as n4 first confidences) of the same first preset behavior among multiple first confidences (such as n4×n3 first confidences, n4 is an integer greater than 1, and n4 is the number of image blocks included in the image block sequence) corresponding to each image block in the image block sequence, and obtain the mean of the first confidences corresponding to each first preset behavior (such as the mean of n3 first confidences). Thereafter, the present disclosure determines the behavior of the target object based on the mean of the first confidences corresponding to each first preset behavior (such as n3 first preset behaviors) and the first thresholds corresponding to the multiple first preset behaviors, thereby obtaining the behavior type to which the behavior of the target object belongs.

[0106] More specifically, for any first preset behavior, the mean of the first confidences of all the first preset behaviors among the multiple first confidences corresponding to each image block is calculated, so as to obtain the first confidence mean of the first preset behavior, and the first confidence mean represents the probability that the behavior of the target object in multiple video frames belongs to the first preset behavior. After calculating the first confidence mean for each first preset behavior, the present disclosure can select the first confidence mean with the highest value from all the first confidence means (such as n3 first confidence means), and determine whether the relationship between the first confidence mean with the highest value and a first threshold value (such as 0.75 or 0.7, etc.) satisfies a predetermined condition (such as determining whether the first confidence mean with the highest value reaches the first threshold value corresponding to the first preset behavior corresponding to the first confidence mean with the highest value, etc.). If the predetermined condition is met, it is determined that the behavior of the target object in multiple video frames is the first preset behavior corresponding to the first confidence mean with the highest value. Assume behavior; if the predetermined condition is not met, it can be determined whether the size relationship between the second highest first confidence mean and another first threshold satisfies the predetermined condition (such as determining whether the second highest first confidence mean reaches the first threshold corresponding to the first preset behavior corresponding to the second highest first confidence mean, etc.), if the predetermined condition is met, it is determined that the behavior of the target object in multiple video frames is the first preset behavior corresponding to the second highest first confidence mean, and so on. If the size relationship between all first confidence means and the corresponding first threshold does not meet the predetermined condition, the behavior of the target object in multiple video frames is not successfully identified this time.

[0107] In another example, the present disclosure may perform weighted average calculation processing on the first confidences (such as n4 first confidences) of the same first preset behavior in multiple first confidences (such as n4×n3 first confidences, n4 is an integer greater than 1, and n4 is the number of image blocks included in the image block sequence) corresponding to each image block in the image block sequence according to the weights corresponding to each image block in the image block sequence (i.e., the weights corresponding to each video frame), so as to obtain the first confidence weighted values ​​corresponding to each first preset behavior (a total of n3 first confidence weighted values). Thereafter, the present disclosure determines the behavior of the target object according to the first confidence weighted values ​​corresponding to each first preset behavior (such as n3 first preset behaviors) and the first thresholds corresponding to the multiple first preset behaviors, so as to obtain the behavior type to which the behavior of the target object belongs.

[0108] More specifically, for any first preset behavior, the weighted average of the first confidences of all the first preset behaviors among the multiple first confidences corresponding to each image block is calculated, so as to obtain the first confidence weighted value of the first preset behavior, and the first confidence weighted value represents the probability that the behavior of the target object in multiple video frames belongs to the first preset behavior. After performing the first confidence weighted average calculation for each first preset behavior, the present disclosure can select the first confidence weighted value with the highest value from all the first confidence weighted values ​​(such as n3 first confidence weighted values), and determine whether the magnitude relationship between the first confidence weighted value with the highest value and a first threshold value (such as 0.75 or 0.7, etc.) satisfies a predetermined condition (such as determining whether the first confidence weighted value with the highest value reaches the first threshold value corresponding to the first preset behavior corresponding to the first confidence weighted value with the highest value, etc.). If the predetermined condition is met, it is determined that the behavior of the target object in multiple video frames is the first confidence weighted value corresponding to the first preset behavior with the highest value. a preset behavior; if the predetermined condition is not met, it can be determined whether the size relationship between the second highest first confidence weighted value and another first threshold satisfies the predetermined condition (such as determining whether the second highest first confidence weighted value reaches the first threshold corresponding to the first preset behavior corresponding to the second highest first confidence weighted value, etc.); if the predetermined condition is met, it is determined that the behavior of the target object in the multiple video frames is the first preset behavior corresponding to the second highest first confidence weighted value, and so on. If the size relationship between all the first confidence weighted values ​​and the corresponding first thresholds does not meet the predetermined condition, the behavior of the target object in the multiple video frames is not successfully identified this time.

[0109] Optionally, the weights corresponding to each video frame can be set according to the acquisition time of each video frame. For example, the weight corresponding to the video frame with an earlier acquisition time is not less than the weight corresponding to the video frame with a later acquisition time. Because human behavior is sometimes more intense at the beginning, and the behavior will gradually weaken in the subsequent stages, the present disclosure sets the weights corresponding to each video frame according to the acquisition time of each video frame. In fact, the weights corresponding to each video frame are set based on the attenuation of behavior over time, which is conducive to improving the accuracy of the confidence corresponding to each first preset behavior finally determined by the present disclosure. In addition, when setting the weights corresponding to each video frame, factors such as the image clarity of the video frame and the degree of facial deflection of the target object in the video frame can also be considered, which is conducive to avoiding the influence of video frames with poor clarity or faces with large angle deflection on the accuracy of multiple first confidences corresponding to each first preset behavior finally determined.

[0110] The present disclosure determines the behavior of the target object in multiple video frames by utilizing multiple confidence levels corresponding to multiple image blocks, thereby achieving smooth processing of the behavior of the target object in multiple video frames, which is not only beneficial to avoiding the inaccurate phenomenon of determining the behavior of the target object using a single video frame, but also beneficial to avoiding the phenomenon of behavior recognition jumps, thereby helping to improve the accuracy of behavior recognition of the target object.

[0111] In an optional example, if the behavior type to which the behavior of the target object belongs conforms to the second behavior type, the present disclosure may use the first preset behavior corresponding to the first confidence (such as the first confidence mean or the first confidence weighted value) whose magnitude relationship with the first threshold satisfies a predetermined condition as the behavior of the target object. In other words, if the magnitude relationship between the first confidence of a first preset behavior belonging to the second behavior type and the first threshold corresponding to the first preset behavior satisfies a predetermined condition, the behavior of the target object is identified as the first preset behavior.

[0112] Optionally, the second behavior type in the present disclosure may refer to a type that can very positively determine that the behavior of the target object is a specific behavior (such as dangerous behavior or other behaviors that need to be identified). The second behavior type may be referred to as determining a predetermined behavior type (such as determining a dangerous behavior type, etc.). Under normal circumstances, the second behavior type in the present disclosure may include: multiple first preset behaviors. For example, the second behavior type may include: smoking behavior and non-smoking behavior. For another example, the second behavior type may include: calling behavior and non-calling behavior, etc.

[0113] To give a more specific example, assume that there are three first preset behaviors, namely: smoking behavior, suspected smoking behavior and non-smoking behavior, among which smoking behavior and non-smoking behavior belong to the second behavior type. If the three confidence levels finally obtained by the present disclosure for the above three first preset behaviors are: 0.8, 0.1 and 0.1, respectively, then since 0.8 reaches the first threshold corresponding to the smoking behavior (such as 0.75 or 0.7, etc.), the present disclosure can determine that the behavior of the target object in multiple video frames is smoking behavior.

[0114] The technical solution provided by the present disclosure can set the first threshold of specific behaviors that need to be identified, such as smoking or making phone calls, to be slightly higher, thereby minimizing the false alarm phenomenon of specific behaviors that need to be identified, such as dangerous behaviors; because the present disclosure adopts a cascade behavior recognition method in the process of identifying the behavior of the target object, and the cascade behavior recognition method can make up for the phenomenon of missed recognition of specific behaviors caused by the slightly high setting of the first threshold, therefore, the present disclosure effectively reduces the difficulty of setting the threshold, which is conducive to improving the ease of use of behavior recognition.

[0115] In an optional example, the present disclosure determines an example of a behavior of the target object according to a result of performing behavior recognition processing on the image block sequence for a second preset behavior. Fig. 9 shown.

[0116] Fig. 9 In step S900, a feature vector extraction process based on a second preset behavior is performed on the image block sequence to obtain a second feature vector corresponding to the image block sequence.

[0117] Optionally, the feature vector extraction process based on the second preset behavior in the present disclosure is a feature vector extraction method, and the feature vector extraction method is a feature vector extraction method set for the purpose of identifying the second preset behavior. For example, the present disclosure can use a neural network to implement feature vector extraction processing, and the present disclosure can use training samples with second preset behavior annotation information to train the neural network. After the neural network is successfully trained, the network parameters of the neural network are obtained, and the feature vector extraction process performed by the neural network using the network parameters is the feature vector extraction process based on the second preset behavior.

[0118] An example of a process of using a neural network to extract a feature vector based on a second preset behavior for an image block sequence in the present disclosure may be: each image block in the image block sequence is respectively processed by a fourth neural network for extracting features, and a feature vector (such as a one-dimensional array formed by multiple vector elements for describing the characteristics of the target part) of each image block in the image block sequence is obtained based on the processing result of the fourth neural network, and the feature vectors of all image blocks are the second feature vectors corresponding to the image block sequence.

[0119] The fourth neural network in the present disclosure may be a convolutional neural network, for example, the fourth neural network may also include VargNet. The fourth neural network and the second neural network may have the same network structure, but usually have different network parameters. The present disclosure may use one fourth neural network to obtain the feature vector of each image block, or may use multiple fourth neural networks to simultaneously obtain the feature vector of each image block. Multiple fourth neural networks usually have the same network structure and the same network parameters.

[0120] S901. For any image block in an image block sequence, determine, according to a second feature vector of the image block, confidence levels that behaviors of a target object are at least two second preset behaviors, to obtain at least two second confidence levels.

[0121] Optionally, the present disclosure may successively perform calculations on the first feature vectors of each image block through a fifth neural network for behavior classification, and obtain multiple second confidences corresponding to each image block according to the calculation results of the fifth neural network. For example, the second preset behavior includes n5 (n5 is an integer greater than 1) behaviors (i.e., the number of second preset behaviors is n5, for example, the behavior of using a straw and the behavior of not using a straw, two second preset behaviors; another example, the behavior of using an instant message and the behavior of not using an instant message, two second preset behaviors), and the n5 second confidences corresponding to each image block are a group of second confidences.

[0122] Optionally, the fifth neural network and the third neural network may have the same network structure, but usually have different network parameters. The fifth neural network and the third neural network usually both include at least one layer for classification (such as a fully connected layer, etc.). In addition, the fifth neural network is usually combined with the fourth neural network and trained together.

[0123] In an optional example, for any image block, the present disclosure can not only use the second feature vector of the image block to determine the second confidence that the behavior of the target object is at least two second preset behaviors, but also use the second feature vector and the first confidence of the image block (such as the first confidence corresponding to the image block, and for example, the first confidence corresponding to the image block sequence) to determine the confidence that the behavior of the target object is at least two second preset behaviors.

[0124] In one example, first, for any image block in an image block sequence, the present disclosure can fuse multiple first confidences corresponding to the image block and the second feature vector of the image block (such as using an encoder to encode multiple first confidences corresponding to the image block, and concatenating the encoding result and the second feature vector), thereby obtaining a fused feature vector of the image block. Secondly, the present disclosure can perform calculations on the fused feature vectors of each image block respectively through a fifth neural network, and obtain the confidences of at least two second preset behaviors of the behavior of the target object in each image block according to the calculation results of the fifth neural network, that is, obtain at least two second confidences corresponding to each image block.

[0125] In another example, first, the present disclosure can fuse multiple first confidences corresponding to the image block sequence with the second feature vectors of each image block in the image sequence (such as using an encoder to encode multiple first confidences corresponding to the image block sequence, and splicing the encoding results and the second feature vectors of each image block), thereby obtaining a fused feature vector of each image block. Secondly, the present disclosure can perform calculations on the fused feature vectors of each image block through a fifth neural network, and obtain the confidence that the behavior of the target object in each image block is at least two second preset behaviors according to the calculation results of the fifth neural network, that is, obtain at least two second confidences corresponding to each image block.

[0126] The present disclosure facilitates the acquisition of more comprehensive information by fusing the first confidence level with the second feature vector in the computational processing of the fifth neural network, thereby facilitating the improvement of the accuracy of the computational results of the fifth neural network, that is, improving the accuracy of behavior classification.

[0127] S902: Determine a behavior of the target object according to at least two second confidences corresponding to each image block in the image block sequence and second thresholds corresponding to at least two second preset behaviors.

[0128] Optionally, the present disclosure may perform corresponding calculation processing on the multiple second confidences corresponding to each image block, so as to obtain multiple second confidences (such as n5 confidences) corresponding to the image block sequence. The present disclosure may determine whether the size relationship between the multiple second confidences corresponding to the image block sequence and the second threshold value meets a predetermined condition, and determine the behavior of the target object in the multiple video frames according to the judgment result. For example, assuming that the second preset behavior includes: a first behavior (such as using a straw behavior or using an instant message behavior, etc.) and a second behavior (such as not using a straw behavior or not using an instant message behavior, etc.), if at least two second confidences corresponding to each image block in the image block sequence and the second threshold corresponding to the first behavior in the multiple second preset behaviors meet the first preset condition, then the behavior of the target object is determined to be the first behavior. If at least two second confidences corresponding to each image block in the image block sequence and the second threshold corresponding to the second behavior in the multiple second preset behaviors meet the second preset condition, then the behavior of the target object is determined to be a specific behavior in the multiple first preset behaviors, such as smoking behavior or making a phone call behavior.

[0129] When the present disclosure identifies that the behavior of the target object is not the first behavior, it can be clearly known that the behavior of the target object is not a suspected specific behavior (such as suspected smoking behavior or suspected phone call behavior, etc.). At this time, the behavior of the target object is most likely a specific behavior in the first preset behavior (such as smoking behavior or phone call behavior, etc.). Therefore, the present disclosure determines the behavior of the target object through the results of the second-level behavior recognition, which is conducive to improving the accuracy of behavior recognition.

[0130] In one example, the present disclosure may calculate the mean of the second confidences (such as n4 second confidences) of the same second preset behavior among multiple second confidences (such as n5×n4 second confidences, n5 is an integer greater than 1, and n4 is the number of image blocks included in the image block sequence) corresponding to each image block in the image block sequence, and obtain the mean of the second confidences corresponding to each second preset behavior (such as the mean of n5 second confidences). Thereafter, the present disclosure determines the behavior of the target object based on the mean of the second confidences corresponding to each second preset behavior (such as n5 second preset behaviors) and the second thresholds corresponding to the multiple second preset behaviors.

[0131] More specifically, for any second preset behavior, the mean of the second confidences of all the second preset behaviors among the multiple second confidences corresponding to each image block is calculated to obtain the second confidence mean of the second preset behavior, and the second confidence mean represents the probability that the behavior of the target object in multiple video frames belongs to the second preset behavior. After calculating the second confidence mean for each second preset behavior, the present disclosure can select the second confidence mean with the highest value from all the second confidence means (such as n5 second confidence means), and determine whether the relationship between the second confidence mean with the highest value and a second threshold value (such as 0.75 or 0.7, etc.) satisfies a predetermined condition (such as determining whether the second confidence mean with the highest value reaches the second threshold value corresponding to the second preset behavior corresponding to the second confidence mean with the highest value, etc.). If the predetermined condition is met, it is determined that the behavior of the target object in multiple video frames is the second preset behavior corresponding to the second confidence mean with the highest value. Assume behavior; if the predetermined condition is not met, it can be determined whether the size relationship between the second confidence mean with the second highest value and another second threshold satisfies the predetermined condition (such as determining whether the second confidence mean with the second highest value reaches the second threshold corresponding to the second preset behavior corresponding to the second confidence mean with the second highest value, etc.), if the predetermined condition is met, it is determined that the behavior of the target object in multiple video frames is the second preset behavior corresponding to the second confidence mean with the second highest value, and so on. If the size relationship between all second confidence means and the corresponding second thresholds does not meet the predetermined condition, the behavior of the target object in multiple video frames is not successfully identified this time.

[0132] In another example, the present disclosure may perform weighted average calculation processing on the second confidences (such as n4 second confidences) of the same second preset behavior in multiple second confidences (such as n5×n4 first confidences, n5 is an integer greater than 1, and n4 is the number of image blocks included in the image block sequence) corresponding to each image block in the image block sequence according to the weights corresponding to each image block in the image block sequence (i.e., the weights corresponding to each video frame), so as to obtain the second confidence weighted values ​​corresponding to each second preset behavior (a total of n3 first confidence weighted values). Thereafter, the present disclosure determines the behavior of the target object according to the second confidence weighted values ​​corresponding to each second preset behavior (such as n5 second preset behaviors) and the second threshold values ​​corresponding to each second preset behavior.

[0133] More specifically, for any second preset behavior, the weighted average of the second confidences of all the second preset behaviors among the multiple second confidences corresponding to each image block is calculated to obtain the second confidence weighted value of the second preset behavior, and the second confidence weighted value represents the probability that the behavior of the target object in multiple video frames belongs to the second preset behavior. After performing the second confidence weighted average calculation for each second preset behavior, the present disclosure can select the second confidence weighted value with the highest value from all the second confidence weighted values ​​(such as n5 second confidence weighted values), and determine whether the magnitude relationship between the second confidence weighted value with the highest value and a second threshold value (such as 0.75 or 0.7, etc.) satisfies a predetermined condition (such as determining whether the second confidence weighted value with the highest value reaches the second threshold value corresponding to the second preset behavior corresponding to the second confidence weighted value with the highest value, etc.). If the predetermined condition is met, it is determined that the behavior of the target object in multiple video frames is the second preset behavior corresponding to the second confidence weighted value with the highest value. Second preset behavior; if the predetermined condition is not met, it can be determined whether the size relationship between the second highest second confidence weighted value and another second threshold value meets the predetermined condition (such as determining whether the second highest second confidence weighted value reaches the second threshold value corresponding to the second preset behavior corresponding to the second highest second confidence weighted value, etc.), if the predetermined condition is met, it is determined that the behavior of the target object in multiple video frames is the second preset behavior corresponding to the second highest second confidence weighted value, and so on. If the size relationship between all second confidence weighted values ​​and the corresponding second threshold value does not meet the predetermined condition, the behavior of the target object in multiple video frames is not successfully identified this time. The setting method of the weights corresponding to each video frame can refer to the relevant description in the above embodiment, and will not be described in detail here.

[0134] The present disclosure determines the behavior of the target object in multiple video frames by utilizing multiple second confidences corresponding to multiple image blocks, thereby achieving smooth processing of the behavior of the target object in multiple video frames, which is not only beneficial to avoid the inaccurate phenomenon of determining the behavior of the target object using a single video frame, but also beneficial to avoid the phenomenon of behavior recognition jumps, thereby helping to improve the accuracy of behavior recognition.

[0135] Exemplary Devices

[0136] Fig.10 1 is a schematic diagram of the structure of an embodiment of a behavior recognition device of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure.

[0137] like Fig.10The device shown includes: a sequence acquisition module 1000, a feature vector acquisition module 1001, a behavior type determination module 1002, and a cascade recognition module 1003. The device may also optionally include: a behavior recognition module 1004.

[0138] The sequence acquisition module 1000 is used to acquire an image block sequence including a target part of a target object from a plurality of video frames acquired by a camera device disposed in a mobile device.

[0139] The feature vector acquisition module 1001 is used to perform feature vector extraction processing based on a first preset behavior on the image block sequence acquired by the sequence acquisition module 10000 to obtain a first feature vector corresponding to the image block sequence.

[0140] The behavior type determination module 1002 is used to determine the behavior type to which the behavior of the target object belongs according to the first feature vector obtained by the feature vector acquisition module 1001 .

[0141] The cascade recognition module 1003 is used to determine the behavior of the target object according to the result of the behavior recognition processing of the image block sequence for the second preset behavior if the behavior type determined by the behavior type determination module 1002 matches the first behavior type.

[0142] Optionally, the acquisition sequence module 1000 may include: a first submodule 10001, a second submodule 10002, and a third submodule 10003. The first submodule 10001 is used to obtain an image region containing at least one target part of a target object in any video frame among a plurality of video frames acquired from a camera device disposed in a mobile device, and obtain at least one image region. The second submodule 10002 is used to determine the target region of the video frame according to the at least one image region obtained by the first submodule 10001. The third submodule 10003 is used to obtain an image block containing the target part of the target object in the video frame according to the target region of the video frame determined by the second submodule 10002.

[0143] Optionally, the behavior type determination module 1002 may include: a fourth submodule 10021 and a fifth submodule 10022. The fourth submodule 10021 may be used to determine the confidences of the behaviors of the target object as multiple first preset behaviors for any image block in the image block sequence, based on the first feature vector of the image block obtained by the feature vector acquisition module 1001, and obtain multiple first confidences. The fifth submodule 10022 is used to determine the behavior type to which the behavior of the target object belongs, based on the multiple first confidences corresponding to each image block in the image block sequence and the first thresholds corresponding to the multiple first preset behaviors. At least one of the multiple first preset behaviors belongs to the first behavior type, and at least one of the multiple first preset behaviors belongs to the second behavior type.

[0144] Optionally, the fifth submodule 10022 can be further used to calculate the mean of the first confidences of the same first preset behavior among the multiple first confidences corresponding to each image block in the image block sequence, obtain the means of the first confidences corresponding to the multiple first preset behaviors, and determine the behavior type to which the behavior of the target object belongs based on the means of the first confidences corresponding to the multiple first preset behaviors and the first thresholds corresponding to the multiple first preset behaviors.

[0145] Optionally, the fifth submodule 10022 can be further used to perform weighted average calculation on the first confidence of the same first preset behavior among the multiple first confidences corresponding to each image block in the image block sequence according to the weights corresponding to each image block in the image block sequence, obtain the first confidence weighted values ​​corresponding to each of the multiple first preset behaviors, and determine the behavior type to which the behavior of the target object belongs according to the first confidence weighted values ​​corresponding to each of the multiple first preset behaviors and the first thresholds corresponding to each of the multiple first preset behaviors. The weight corresponding to the image block in the video frame with a later acquisition time is not higher than the weight corresponding to the image block in the video frame with an earlier acquisition time.

[0146] Optionally, the behavior identification module 1004 is used to determine that if the behavior type of the target object's behavior determined by the behavior type module 1002 meets the second behavior type, the first preset behavior corresponding to the first confidence level whose size relationship with the first threshold satisfies a predetermined condition is used as the behavior of the target object.

[0147] Optionally, the cascade recognition module 1003 includes: a sixth submodule 10031, a seventh submodule 10032 and an eighth submodule 10033. The sixth submodule 10031 is used to perform feature vector extraction processing based on the second preset behavior on the image block sequence to obtain the second feature vector corresponding to the image block sequence. The seventh submodule 10032 is used to determine the confidence of the behavior of the target object as at least two second preset behaviors for any image block in the image block sequence according to the second feature vector of the image block, and obtain at least two second confidences. The eighth submodule 10033 is used to determine the behavior of the target object according to at least two second confidences corresponding to each image block in the image block sequence and second thresholds corresponding to at least two second preset behaviors.

[0148] Optionally, the seventh submodule 10032 is further used to fuse multiple first confidences corresponding to any image block in the image block sequence and the second feature vector of the image block to obtain a fused feature vector of the image block; and determine, based on the fused feature vectors of each image block in the image block sequence, that the behavior of the target object is the confidence of at least two second preset behaviors, to obtain at least two second confidences.

[0149] Optionally, the eighth submodule 10033 is further used to determine that the behavior of the target object is the first behavior if at least two second confidences corresponding to each image block in the image block sequence and the second threshold corresponding to the first behavior among the multiple second preset behaviors meet the first preset condition; if at least two second confidences corresponding to each image block in the image block sequence and the second threshold corresponding to the second behavior among the multiple second preset behaviors meet the second preset condition, then determine that the behavior of the target object is a first preset behavior among the multiple first preset behaviors.

[0150] Exemplary Electronic Devices

[0151] Reference below Fig.11 An electronic device according to an embodiment of the present disclosure is described. Fig.11 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Fig.11 As shown, the electronic device 111 includes one or more processors 1111 and a memory 1112 .

[0152] The processor 1111 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 111 to perform desired functions.

[0153] The memory 1112 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, may include: random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory, for example, may include: read-only memory (ROM), hard disk, and flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1111 may run the program instructions to implement the behavior recognition method of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0154] In one example, the electronic device 111 may also include: an input device 1113 and an output device 1114, etc., which are interconnected through a bus system and / or other forms of connection mechanisms (not shown). In addition, the input device 1113 may also include, for example, a keyboard, a mouse, etc. The output device 1114 may output various information to the outside. The output device 1114 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0155] Of course, to simplify, Fig.11 Only some of the components related to the present disclosure in the electronic device 111 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 111 may also include any other appropriate components.

[0156] Exemplary computer program products and computer-readable storage media

[0157] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the behavior recognition method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0158] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0159] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the behavior recognition method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0160] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive enumeration) of readable storage media can include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0161] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0162] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0163] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are intended only as illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, and configured in any manner. Words such as "include," "comprise," "have," and the like are open words, meaning "including but not limited to," and may be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.

[0164] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0165] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0166] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects, etc., will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0167] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A behavior recognition method, include: Acquire a sequence of image blocks including a target part of a target object from a plurality of video frames acquired by a camera device disposed in a mobile device, wherein the mobile device comprises a vehicle capable of carrying people; Performing feature vector extraction processing based on a first preset behavior on the image block sequence to obtain a first feature vector corresponding to the image block sequence; Determining, according to the first feature vector, a behavior type to which the behavior of the target object belongs; If the behavior type to which the behavior of the target object belongs conforms to the first behavior type, the behavior of the target object is determined based on the result of behavior recognition processing of the image block sequence for the second preset behavior, the first behavior type is a suspected specific behavior type, and the second preset behavior is a behavior similar to the specific behavior included in the first preset behavior.

2. The method according to claim 1, in, The step of acquiring a sequence of image blocks containing a target part of a target object from a plurality of video frames acquired by a camera device disposed in the mobile device comprises: For any one of the plurality of video frames acquired from the camera device disposed in the mobile device, an image region containing at least one target part of the target object in the video frame is acquired to obtain at least one image region; Determining a target area of ​​the video frame according to the at least one image area; According to the target area of ​​the video frame, an image block of the video frame including the target part of the target object is obtained.

3. The method according to claim 1, in, The determining, according to the first feature vector, a behavior type to which the behavior of the target object belongs includes: For any image block in the image block sequence, according to the first feature vector of the image block, determine the confidences that the behavior of the target object is a plurality of first preset behaviors, and obtain a plurality of first confidences; according to the plurality of first confidences corresponding to each image block in the image block sequence and the first thresholds corresponding to the plurality of first preset behaviors, determine the behavior type to which the behavior of the target object belongs; At least one first preset behavior among the plurality of first preset behaviors belongs to a first behavior type, and at least one first preset behavior among the plurality of first preset behaviors belongs to a second behavior type.

4. The method according to claim 3, in, The determining, according to the plurality of first confidences corresponding to the respective image blocks in the image block sequence and the first thresholds corresponding to the respective first preset behaviors, the behavior type to which the behavior of the target object belongs includes: Calculating the mean of first confidences of the same first preset behavior among the multiple first confidences corresponding to each image block in the image block sequence to obtain the first confidence mean values ​​corresponding to the multiple first preset behaviors, and determining the behavior type to which the behavior of the target object belongs according to the first confidence mean values ​​corresponding to the multiple first preset behaviors and the first threshold values ​​corresponding to the multiple first preset behaviors; or performing weighted average calculation on first confidences of the same first preset behavior among a plurality of first confidences corresponding to the respective image blocks in the image block sequence according to the weights respectively corresponding to the respective image blocks to obtain first confidence weighted values ​​respectively corresponding to the plurality of first preset behaviors, and determining the behavior type to which the behavior of the target object belongs according to the first confidence weighted values ​​respectively corresponding to the plurality of first preset behaviors and the first thresholds respectively corresponding to the plurality of first preset behaviors; The weight corresponding to the image block in the video frame acquired at a later time is not higher than the weight corresponding to the image block in the video frame acquired at an earlier time.

5. The method according to claim 3, in, The method further comprises: If the behavior type to which the behavior of the target object belongs conforms to the second behavior type, a first preset behavior corresponding to a first confidence level whose magnitude relationship with the first threshold satisfies a predetermined condition is taken as the behavior of the target object.

6. The method according to any one of claims 3 to 5, in, The determining the behavior of the target object according to the result of the behavior recognition processing of the second preset behavior on the image block sequence includes: Performing feature vector extraction processing based on a second preset behavior on the image block sequence to obtain a second feature vector corresponding to the image block sequence; For any image block in the image block sequence, determining, according to the second feature vector of the image block, the confidence levels of the behaviors of the target object being at least two second preset behaviors, and obtaining at least two second confidence levels; The behavior of the target object is determined according to at least two second confidences corresponding to each image block in the image block sequence and second thresholds corresponding to each of the at least two second preset behaviors.

7. The method according to claim 6, in, The step of determining, for any image block in the image block sequence, the confidence levels of the behaviors of the target object as at least two second preset behaviors according to the second feature vector of the image block, and obtaining at least two second confidence levels comprises: For any image block in the image block sequence, a fusion process is performed on a plurality of first confidences corresponding to the image block and a second feature vector of the image block to obtain a fused feature vector of the image block; According to the fused feature vectors of each image block in the image block sequence, it is determined that the behaviors of the target object are confidence levels of at least two second preset behaviors, and at least two second confidence levels are obtained.

8. A behavior recognition device, include: An acquisition sequence module is used to acquire an image block sequence containing a target part of a target object from a plurality of video frames acquired by a camera device disposed in a mobile device, wherein the mobile device comprises a vehicle capable of carrying people; a feature vector acquisition module, configured to perform feature vector extraction processing based on a first preset behavior on the image block sequence acquired by the sequence acquisition module, and obtain a first feature vector corresponding to the image block sequence; and a behavior type determination module, configured to determine the behavior type to which the behavior of the target object belongs according to the first feature vector acquired by the feature vector acquisition module; A cascade recognition module is used to determine the behavior of the target object based on the result of behavior recognition processing of the image block sequence for a second preset behavior if the behavior type determined by the behavior type determination module matches the first behavior type, the first behavior type is a suspected specific behavior type, and the second preset behavior is a behavior similar to the specific behavior included in the first preset behavior.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 7.

10. An electronic device, wherein the electronic device include: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 7.

Images