A multi-data fusion behavior recognition method and system based on deep learning

Abstract

The present invention provides a multi-data fusion behavior recognition method and system based on deep learning, which relates to the field of image processing technology. The present invention first performs initial behavior recognition on monitoring video data through a behavior model recognition unit, which can quickly detect suspected behavior of making a phone call; then, the focal length parameter of the monitoring video acquisition device is adjusted through a target image extraction unit to extract personnel area images, human skeleton data, hand area images and mobile phone target images from the monitoring video data, thereby ensuring that the image clarity is available and the detection range is reasonable; finally, the monitoring video data is subjected to secondary behavior recognition through a behavior logic recognition unit, and by judging whether the behavior of the target person conforms to the behavior logic of making and receiving calls, it is determined whether he has the behavior of making and receiving calls, thereby ensuring the accuracy of recognition.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-data fusion behavior recognition method and system based on deep learning. Background Art

[0002] In some on-stage scenarios (such as conference discussions, teacher lectures, etc.), rashly answering or making phone calls is a very uncivilized behavior, which will interrupt the logical process of the audience and greatly affect the knowledge teaching effect or the efficiency of conference communication. Therefore, in some on-stage scenarios, we need to identify and discover such uncivilized behavior as answering or making phone calls.

[0003] Most existing behavior recognition systems use general behavior recognition models. Although such general behavior recognition models are easy to obtain and can recognize a variety of different behaviors, when it comes to identifying uncivilized behaviors such as making phone calls, the behavior recognition accuracy of general behavior recognition models is seriously insufficient, which is reflected not only in missed behaviors but also in false behaviors.

[0004] Missed detection will only affect the accuracy of behavioral assessment, while after a behavioral misdetection occurs, the wrong recognition result will cause unnecessary trouble to the examinee and even cause psychological burden to the examinee; therefore, we need to do our utmost to prevent the occurrence of false detection of phone calls; however, general behavior recognition models often misdetect behaviors such as holding a microphone, laser pen, or blackboard brush as making or receiving a phone call; and when only holding a mobile phone or other tools with partial occlusion, they will also be misdetected as making or receiving a phone call; it cannot meet the urgent need to reduce false detections.

[0005] Therefore, a multi-data fusion behavior recognition method and system based on deep learning is needed to solve the above technical problems. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a multi-data fusion behavior recognition method and system based on deep learning, which mainly solve the problems of the prior art: 1. It is easy to make false detections when only using a general behavior recognition model to judge whether to make or receive a call; 2. If more false detection judgment mechanisms are introduced, the detection efficiency will not keep up; 3. The monitoring images collected by the monitoring are not clear enough and the image detection range is too large.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] On the one hand, a multi-data fusion behavior recognition system based on deep learning is provided, including a monitoring video acquisition device, a target image extraction unit, a behavior model recognition unit and a behavior logic judgment unit; wherein,

[0009] Surveillance video acquisition equipment: used to collect surveillance video data within the surveillance area;

[0010] Target image extraction unit: used to extract mobile phone target images from surveillance video data; including personnel image extraction module, skeleton data extraction module, hand image extraction module and mobile phone target extraction module; wherein, the personnel image extraction module is used to extract personnel area images from surveillance video data, the skeleton data extraction module is used to extract human skeleton data from personnel area images, the hand image extraction module extracts hand area images from surveillance video data based on human skeleton data, and the mobile phone target extraction module is used to extract mobile phone target images from hand area images;

[0011] Behavior model recognition unit: used for initial behavior recognition of surveillance video data; includes a multi-data fusion database, a dedicated behavior recognition model and a model training management module; wherein the multi-data fusion database is used to store image training data, the dedicated behavior recognition model is used to perform initial behavior recognition, and the model training management module is used to train and manage the dedicated behavior recognition model;

[0012] Behavior logic recognition unit: used to perform secondary behavior recognition on surveillance video data; includes an eye image extraction module, an ear image extraction module and a behavior logic recognition module; wherein the eye image extraction module is used to extract the eye gaze direction from the personnel area image, the ear image extraction module is used to extract the ear image position from the personnel area image, and the behavior logic recognition module is used to perform secondary behavior recognition.

[0013] On the other hand, a multi-data fusion behavior recognition method based on deep learning is provided, which is applied to the multi-data fusion behavior recognition system based on deep learning as described above, and the behavior of making and receiving mobile phone calls is recognized through the following steps:

[0014] Step 1: Continuously collect surveillance video data within the surveillance area through surveillance video acquisition equipment;

[0015] Step 2: Continuously perform initial behavior recognition on the surveillance video data through the behavior model recognition unit; if the initial behavior recognition is: suspected phone call behavior, then start step 4;

[0016] Step 3: extracting the mobile phone target image from the surveillance video data through the target image extraction unit; if the mobile phone target image is successfully extracted, then start executing step 4; otherwise, return to executing step 1;

[0017] Step 4: Perform secondary behavior recognition on the surveillance video data through the behavior logic recognition unit; if the secondary behavior recognition is: the logic of making or receiving a phone call is established, it is determined that there is a behavior of making or receiving a phone call; otherwise, it is determined that there is no behavior of making or receiving a phone call;

[0018] Step 5: Return to step 1 until the identification of the phone call and answering behavior is completed.

[0019] As a further solution, the behavior logic recognition unit determines whether there is a call making behavior through the following steps:

[0020] Step 41: Determine the target position of the mobile phone by positioning the mobile phone target image;

[0021] Step 42: using the eye image extraction module to determine whether the target person's eyes are looking at the target position direction of the mobile phone; if so, proceed to step 44; if not, proceed to step 43;

[0022] Step 43: using the ear image extraction module to determine whether the ear of the target person coincides with the target position of the mobile phone; if so, executing step 44; if not, determining that there is no phone call behavior;

[0023] Step 44: continuously record the direction of the eyes looking at the target position of the mobile phone or the overlap time of the ear at the target position of the mobile phone; if the time threshold for determining whether to make or receive a call is reached, it is determined that there is a call making behavior; otherwise, it is determined that there is no call making behavior.

[0024] As a further solution, the personnel image extraction module extracts the personnel area image through the following steps:

[0025] Acquire surveillance video data and perform frame extraction operations to obtain a number of surveillance image data;

[0026] Perform personnel identification on the surveillance image data and determine whether the identified person is the target person;

[0027] After the target person is found, the target person is selected by the first rectangular area with the target person as the center, and the first rectangular area is continuously shrunk until the selected area is minimized;

[0028] Taking the first rectangular area as a reference, enlarging from the center outward until a preset enlargement factor is reached, thereby obtaining a second rectangular area;

[0029] The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the second rectangular area, and the adjusted surveillance image data is intercepted as the person area image.

[0030] As a further solution, the hand image extraction module extracts the hand area image through the following steps:

[0031] Processing the personnel image data through the skeleton data extraction module to obtain human skeleton data;

[0032] Joint positioning is performed through human skeleton data to obtain elbow point data and wrist point data;

[0033] The elbow point data and the wrist point data are respectively located in the personnel image data to obtain the elbow point and the wrist point;

[0034] According to the preset first point position reasoning relationship, the palm point is inferred through the elbow point and the wrist point, and a third rectangular area is constructed with the palm point as the center;

[0035] The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the third rectangular area, and the adjusted surveillance image data is intercepted as the hand image data.

[0036] As a further solution, when wrist point data cannot be obtained:

[0037] Use human skeleton data to locate joints and obtain shoulder point data and elbow point data;

[0038] The shoulder point data and the elbow point data are respectively located in the personnel image data to obtain the shoulder point and the elbow point;

[0039] According to the preset second point position reasoning relationship, the palm point is inferred through the shoulder point and the elbow point, and a fourth rectangular area is constructed with the palm point as the center;

[0040] The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the fourth rectangular area, and the adjusted surveillance image data is intercepted as the hand image data.

[0041] As a further solution, when the human skeleton data cannot be obtained, the person area image is directly used as the hand area image.

[0042] As a further solution, when there is a phone call, the corresponding person area image, human skeleton data and hand area image are stored in a multi-data fusion database and used as image training data for a dedicated behavior recognition model.

[0043] As a further solution, the behavior model recognition unit takes the person area image as the overall feature data, the hand area image as the local feature data, and the human skeleton data as the associated feature data; by adding a self-attention mechanism to the human skeleton data, the correlation between joint points is established, and by adding an attention mechanism to the person area image, human skeleton data and hand area image and adjusting the weights, multi-data fusion training data is obtained.

[0044] As a further solution, the model training management module is trained in a two-stage training method;

[0045] The first-stage training introduces the EfficientNet sub-model. By training all parameters of the EfficientNet sub-model, the EfficientNet sub-model has preliminary behavior recognition capabilities.

[0046] In the second stage of training, the CNN model is used as the main model. The EfficientNet sub-model is used to perform secondary feature extraction on the person area image, human skeleton data, and hand area image, and these are used as high-quality input features of the main model.

[0047] The main model is trained through high-quality input features, and a CNN model that meets the prediction and recognition accuracy is output as a dedicated behavior recognition model.

[0048] Compared with related technologies, the multi-data fusion behavior recognition method and system based on deep learning provided by the present invention have the following beneficial effects:

[0049] The present invention firstly performs initial behavior recognition on monitoring video data through a behavior model recognition unit, and can quickly discover suspected behavior of making a phone call; then, the focal length parameters of the monitoring video acquisition device are adjusted through a target image extraction unit to extract personnel area images, human skeleton data, hand area images and mobile phone target images from the monitoring video data, thereby ensuring that the image clarity is available and the detection range is reasonable; finally, a behavior logic recognition unit performs secondary behavior recognition on the monitoring video data, and by judging whether the behavior of the target person conforms to the behavior logic of making and receiving calls, it is determined whether he or she has made and received calls, thereby ensuring the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the structure of a multi-data fusion behavior recognition system based on deep learning provided by the present invention;

[0051] Figure 2 A schematic diagram of the steps of a multi-data fusion behavior recognition method based on deep learning provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0053] Example 1

[0054] See also Figure 1 The present invention provides a multi-data fusion behavior recognition system based on deep learning, including a monitoring video acquisition device, a target image extraction unit, a behavior model recognition unit and a behavior logic judgment unit; wherein,

[0055] Surveillance video acquisition equipment: used to collect surveillance video data within the surveillance area;

[0056] Target image extraction unit: used to extract mobile phone target images from surveillance video data; including personnel image extraction module, skeleton data extraction module, hand image extraction module and mobile phone target extraction module; wherein, the personnel image extraction module is used to extract personnel area images from surveillance video data, the skeleton data extraction module is used to extract human skeleton data from personnel area images, the hand image extraction module extracts hand area images from surveillance video data based on human skeleton data, and the mobile phone target extraction module is used to extract mobile phone target images from hand area images;

[0057] Behavior model recognition unit: used for initial behavior recognition of surveillance video data; includes a multi-data fusion database, a dedicated behavior recognition model and a model training management module; wherein the multi-data fusion database is used to store image training data, the dedicated behavior recognition model is used to perform initial behavior recognition, and the model training management module is used to train and manage the dedicated behavior recognition model;

[0058] Behavior logic recognition unit: used to perform secondary behavior recognition on surveillance video data; includes an eye image extraction module, an ear image extraction module and a behavior logic recognition module; wherein the eye image extraction module is used to extract the eye gaze direction from the personnel area image, the ear image extraction module is used to extract the ear image position from the personnel area image, and the behavior logic recognition module is used to perform secondary behavior recognition.

[0059] It should be noted that: the present invention first performs initial behavior recognition on the monitoring video data through the behavior model recognition unit, which can quickly discover suspected mobile phone dialing behavior; then, the focal length parameters of the monitoring video acquisition device are adjusted through the target image extraction unit to extract the personnel area image, human skeleton data, hand area image and mobile phone target image from the monitoring video data, thereby ensuring that the image clarity is available and the detection range is reasonable; finally, the monitoring video data is subjected to secondary behavior recognition through the behavior logic recognition unit, and by judging whether the target person's behavior conforms to the behavior logic of answering and making calls, it is determined whether he has the behavior of answering and making calls, thereby ensuring the accuracy of recognition.

[0060] Example 2

[0061] See also Figure 2The present invention provides a multi-data fusion behavior recognition method based on deep learning, which is applied to a multi-data fusion behavior recognition system based on deep learning as described in Example 1, and recognizes the behavior of making and receiving calls through the following steps:

[0062] Step 1: Continuously collect surveillance video data within the surveillance area through surveillance video acquisition equipment;

[0063] Step 2: Continuously perform initial behavior recognition on the surveillance video data through the behavior model recognition unit; if the initial behavior recognition is: suspected phone call behavior, then start step 4;

[0064] Step 3: extracting the mobile phone target image from the surveillance video data through the target image extraction unit; if the mobile phone target image is successfully extracted, then start executing step 4; otherwise, return to executing step 1;

[0065] Step 4: Perform secondary behavior recognition on the surveillance video data through the behavior logic recognition unit; if the secondary behavior recognition is: the logic of making or receiving a phone call is established, it is determined that there is a behavior of making or receiving a phone call; otherwise, it is determined that there is no behavior of making or receiving a phone call;

[0066] Step 5: Return to step 1 until the identification of the phone call and answering behavior is completed.

[0067] It should be noted that: since the behavior model recognition unit is first used for preliminary recognition, and only after suspicion occurs is further recognized through the target image extraction unit and the behavior logic recognition unit, behavior model detection is more direct and efficient but there are false detections, while logic detection requires adjusting the picture and combining the judgment conditions to be executed, thereby improving detection efficiency and ensuring detection accuracy.

[0068] As a further solution, the behavior logic recognition unit determines whether there is a call making behavior through the following steps:

[0069] Step 41: Determine the target position of the mobile phone by positioning the mobile phone target image;

[0070] Step 42: using the eye image extraction module to determine whether the target person's eyes are looking at the target position direction of the mobile phone; if so, proceed to step 44; if not, proceed to step 43;

[0071] Step 43: using the ear image extraction module to determine whether the ear of the target person coincides with the target position of the mobile phone; if so, executing step 44; if not, determining that there is no phone call behavior;

[0072] Step 44: continuously record the direction of the eyes looking at the target position of the mobile phone or the overlap time of the ear at the target position of the mobile phone; if the time threshold for determining whether to make or receive a call is reached, it is determined that there is a call making behavior; otherwise, it is determined that there is no call making behavior.

[0073] It should be noted that this embodiment creatively introduces behavioral logic recognition into the behavior determination of answering and making phone calls; many times, we only hold a mobile phone or similar items in our hands, but do not actually make or receive phone calls, but the behavior recognition model can easily misdetect such situations. This embodiment first determines whether the target person is holding a mobile phone, and then determines whether he is looking at the phone or answering the phone with his ears, and avoids visual errors in certain frames, by continuously recording the direction of the eye's gaze at the target position of the phone or the overlap time of the ear's target position of the phone, thereby ensuring that the behavior of answering and making phone calls is established.

[0074] As a further solution, the personnel image extraction module extracts the personnel area image through the following steps:

[0075] Acquire surveillance video data and perform frame extraction operations to obtain a number of surveillance image data;

[0076] Perform personnel identification on the surveillance image data and determine whether the identified person is the target person;

[0077] After the target person is found, the target person is selected by the first rectangular area with the target person as the center, and the first rectangular area is continuously shrunk until the selected area is minimized;

[0078] Taking the first rectangular area as a reference, enlarging from the center outward until a preset enlargement factor is reached, thereby obtaining a second rectangular area;

[0079] The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the second rectangular area, and the adjusted surveillance image data is intercepted as the person area image.

[0080] It should be noted that: directly capturing the personnel area image in the original surveillance image data is not clear enough; therefore, this embodiment first finds the target person, and then selects the target person through the first rectangular area, so as to ensure that the target person is selected; the second rectangular area is enlarged (usually enlarged by 10%) to ensure that the edge of the target person is also completely selected. On this basis, instead of directly capturing, the focal length parameters of the magnified surveillance video acquisition device are adjusted to adjust to the second rectangular area to collect the surveillance video data of the target person, and the adjusted surveillance image data is captured as the personnel area image, so as to ensure the clarity and accuracy of the image.

[0081] As a further solution, the hand image extraction module extracts the hand area image through the following steps:

[0082] Processing the personnel image data through the skeleton data extraction module to obtain human skeleton data;

[0083] Joint positioning is performed through human skeleton data to obtain elbow point data and wrist point data;

[0084] The elbow point data and the wrist point data are respectively located in the personnel image data to obtain the elbow point and the wrist point;

[0085] According to the preset first point position reasoning relationship, the palm point is inferred through the elbow point and the wrist point, and a third rectangular area is constructed with the palm point as the center;

[0086] The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the third rectangular area, and the adjusted surveillance image data is intercepted as the hand image data.

[0087] It should be noted that the elbow and wrist points are relatively easy to obtain (the joints are larger); while the palm points are easily blocked and the targets are smaller, so we need to use certain reasoning to obtain the palm points and capture the hand image based on the palm points, so as to better monitor whether the hand is holding a mobile phone.

[0088] The elbow point and wrist point are usually in the same direction and parallel to the palm point, so we determine the palm point through the elbow point and wrist point, and determine the specific selected palm point through a certain proportional relationship (the first point position inference relationship), such as: we determine the end position by 1.5 times the distance from the elbow to the wrist, determine the side length of the third rectangular area by the end position and the wrist point, and construct the third rectangular area with 1 / 2 of the side length as the center (of course, other proportions can be set, the most important core is to determine the palm point through the elbow point and wrist point).

[0089] As a further solution, when wrist point data cannot be obtained:

[0090] Use human skeleton data to locate joints and obtain shoulder point data and elbow point data;

[0091] The shoulder point data and the elbow point data are respectively located in the personnel image data to obtain the shoulder point and the elbow point;

[0092] According to the preset second point position reasoning relationship, the palm point is inferred through the shoulder point and the elbow point, and a fourth rectangular area is constructed with the palm point as the center;

[0093] The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the fourth rectangular area, and the adjusted surveillance image data is intercepted as the hand image data.

[0094] It should be noted that: when this solution cannot accurately identify the wrist point data, a larger fourth rectangular area is selected through the shoulder point and the elbow point to determine the approximate position of the hand. For example, with the elbow point as the center point and 3 times the distance from the shoulder to the elbow as 1 / 2 the length of the square side, the hand image range is predicted (of course, other ratios can also be set, the most important core is to select a larger fourth rectangular area through the shoulder point and the elbow point to determine the approximate position of the hand).

[0095] As a further solution, when the human skeleton data cannot be obtained, the person area image is directly used as the hand area image.

[0096] As a further solution, when there is a phone call, the corresponding person area image, human skeleton data and hand area image are stored in a multi-data fusion database and used as image training data for a dedicated behavior recognition model.

[0097] It should be noted that while we are continuously monitoring and identifying, we are also collecting training image data to continuously iterate the model recognition accuracy and ensure efficient and accurate identification of phone call behaviors.

[0098] As a further solution, the behavior model recognition unit takes the person area image as the overall feature data, the hand area image as the local feature data, and the human skeleton data as the associated feature data; by adding a self-attention mechanism to the human skeleton data, the correlation between joint points is established, and by adding an attention mechanism to the person area image, human skeleton data and hand area image and adjusting the weights, multi-data fusion training data is obtained.

[0099] As a further solution, the model training management module is trained in a two-stage training method;

[0100] The first-stage training introduces the EfficientNet sub-model. By training all parameters of the EfficientNet sub-model, the EfficientNet sub-model has preliminary behavior recognition capabilities.

[0101] In the second stage of training, the CNN model is used as the main model. The EfficientNet sub-model is used to perform secondary feature extraction on the person area image, human skeleton data, and hand area image, and these are used as high-quality input features of the main model.

[0102] The main model is trained through high-quality input features, and a CNN model that meets the prediction and recognition accuracy is output as a dedicated behavior recognition model.

[0103] In a specific embodiment, we divide the hand image data into left and right hand images and perform training through the following training process; wherein,

[0104] 1. Preprocess the personnel area image and left and right hand images and convert them to a size of 256;

[0105] 2. For the personnel area image, the left and right hand images use the secondary full parameter fine-tuning EffiencNet network to extract their 512-dimensional features and normalize them to the 0-1 data space; the skeleton points of the skeleton image (skeleton points, x coordinates, y coordinates, confidence) are normalized from 0-16 to the 0-1 data interval, and the other three values ​​are all data between 0-1. The 17 points are sequentially arranged and merged into a 68-dimensional array. Through the normalization operation, all data are in the 0-1 data interval;

[0106] 3. Since there is a strong dependency between skeleton points, the self-attention mechanism is introduced to extract the associated features between skeleton points through (Q, K, V) calculation. The data after self-attention calculation is upgraded to 512 dimensions;

[0107] 4. Use the attention mechanism to fuse the four groups of 512-dimensional data, and use the adaptive mechanism of attention to complete the weight selection of each input value. The output vector dimension is 512 dimensions. The fused data is normalized to the interval of 0-1; the current 512-dimensional value residual is recorded to prepare for the seventh round of calculation;

[0108] 5. The output linear change of step 4 is increased from 512 dimensions to 1024 dimensions, and the data is 0-1 normalized simultaneously;

[0109] 6. Output the 1024-dimensional vector to 512 dimensions through linear transformation, and perform data 0-1 normalization simultaneously;

[0110] 7. The fifth step data is used as the residual to be fused with the output of the sixth step. During fusion, the coefficient parameter is added to the original residual, and the value after weight fusion is normalized to 0-1;

[0111] 8. Perform a round of 512 linear layer calculations;

[0112] 9. Reduce the 512-dimensional vector to 256 dimensions;

[0113] 10. Add a binary classifier after 256 dimensions (currently two in behavior recognition classification);

[0114] In model training, a two-stage training method is adopted.

[0115] In the first round of training, the CNN model EfficientNet_V2_S was trained on full-parameter classification using the labeled data based on the ImageNet pre-trained model, so that the model has a certain adaptability to the data of the teacher making a phone call scenario.

[0116] The second round of training is to train according to the model designed in the second part of the previous article. In the process of CNN feature extraction of training data, the model trained in the first round is used as the feature extraction model, and its feature correlation is stronger.

[0117] The above are only embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-data fusion behavior recognition method based on deep learning, characterized in that: Use the following steps to identify the behavior of making and receiving calls: Step 1: Continuously collect surveillance video data within the surveillance area through surveillance video acquisition equipment; Step 2: Continuously perform initial behavior recognition on the surveillance video data through the behavior model recognition unit; if the initial behavior recognition is: suspected phone call behavior, then start step 4; Step 3: extracting the mobile phone target image from the surveillance video data through the target image extraction unit; if the mobile phone target image is successfully extracted, then start executing step 4; Otherwise, return to step 1; Step 4: Perform secondary behavior recognition on the surveillance video data through the behavior logic recognition unit; if the secondary behavior recognition is: the logic of making or receiving a phone call is established, it is determined that there is a behavior of making or receiving a phone call; otherwise, it is determined that there is no behavior of making or receiving a phone call; Step 5: Return to step 1 until the identification of the phone call behavior is completed; The behavior logic recognition unit determines whether there is a call making behavior through the following steps: Step 41: Determine the target position of the mobile phone by positioning the target image of the mobile phone; Step 42: using the eye image extraction module to determine whether the target person's eyes are looking at the target position direction of the mobile phone; if so, proceed to step 44; if not, proceed to step 43; Step 43: using the ear image extraction module to determine whether the ear of the target person coincides with the target position of the mobile phone; if so, executing step 44; if not, determining that there is no phone call behavior; Step 44: continuously record the direction of the eye gaze at the target position of the mobile phone or the overlap time of the ear at the target position of the mobile phone; if the time threshold for determining whether to make or receive a call is reached, it is determined that there is a call making behavior; otherwise, it is determined that there is no call making behavior; When there is a phone call, the corresponding person area image, human skeleton data and hand area image are stored in the multi-data fusion database and used as image training data for the dedicated behavior recognition model; The behavior model recognition unit takes the person area image as the overall feature data, the hand area image as the local feature data, and the human skeleton data as the associated feature data; by adding a self-attention mechanism to the human skeleton data, the mutual correlation of joint points is established, and by adding an attention mechanism to the person area image, human skeleton data and hand area image and adjusting the weights, multi-data fusion training data is obtained.

2. According to the multi-data fusion behavior recognition method based on deep learning in claim 1, it is characterized in that: Extract the personnel area image through the following steps: Acquire surveillance video data and perform frame extraction operations to obtain a number of surveillance image data; Perform personnel identification on the surveillance image data and determine whether the identified person is the target person; After the target person is found, the target person is selected by the first rectangular area with the target person as the center, and the first rectangular area is continuously shrunk until the selected area is minimized; Taking the first rectangular area as a reference, enlarging from the center outward until a preset enlargement factor is reached, thereby obtaining a second rectangular area; The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the second rectangular area, and the adjusted surveillance image data is intercepted as the person area image.

3. The multi-data fusion behavior recognition method based on deep learning according to claim 1 is characterized in that: The hand area image is extracted through the following steps: Processing the personnel image data through the skeleton data extraction module to obtain human skeleton data; Joint positioning is performed through human skeleton data to obtain elbow point data and wrist point data; The elbow point data and the wrist point data are respectively located in the personnel image data to obtain the elbow point and the wrist point; According to the preset first point position reasoning relationship, the palm point is inferred through the elbow point and the wrist point, and a third rectangular area is constructed with the palm point as the center; The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the third rectangular area, and the adjusted surveillance image data is intercepted as the hand image data.

4. The multi-data fusion behavior recognition method based on deep learning according to claim 3 is characterized in that: When wrist point data cannot be obtained: Use human skeleton data to locate joints and obtain shoulder point data and elbow point data; The shoulder point data and the elbow point data are respectively located in the personnel image data to obtain the shoulder point and the elbow point; According to the preset second point position reasoning relationship, the palm point is inferred through the shoulder point and the elbow point, and a fourth rectangular area is constructed with the palm point as the center; The focal length parameter of the magnified surveillance video acquisition device is adjusted to collect surveillance video data of the target person in the fourth rectangular area, and the adjusted surveillance image data is intercepted as the hand image data.

5. The multi-data fusion behavior recognition method based on deep learning according to claim 3 is characterized in that: When human skeleton data cannot be obtained, the person area image is directly used as the hand area image.

6. The multi-data fusion behavior recognition method based on deep learning according to claim 1 is characterized in that: The dedicated behavior recognition model is trained using a two-stage training method; The first-stage training introduces the EfficientNet sub-model. By training all parameters of the EfficientNet sub-model, the EfficientNet sub-model has preliminary behavior recognition capabilities. In the second stage of training, the CNN model is used as the main model. The EfficientNet sub-model is used to perform secondary feature extraction on the person area image, human skeleton data, and hand area image, and these are used as high-quality input features of the main model. The main model is trained through high-quality input features, and a CNN model that meets the prediction and recognition accuracy is output as a dedicated behavior recognition model.

7. A multi-data fusion behavior recognition system based on deep learning, using a multi-data fusion behavior recognition method based on deep learning as claimed in any one of claims 1 to 6, characterized in that: It includes monitoring video acquisition equipment, target image extraction unit, behavior model recognition unit and behavior logic judgment unit; wherein, Surveillance video acquisition equipment: used to collect surveillance video data within the surveillance area; Target image extraction unit: used to extract mobile phone target images from surveillance video data; including personnel image extraction module, skeleton data extraction module, hand image extraction module and mobile phone target extraction module; wherein, the personnel image extraction module is used to extract personnel area images from surveillance video data, the skeleton data extraction module is used to extract human skeleton data from personnel area images, the hand image extraction module extracts hand area images from surveillance video data based on human skeleton data, and the mobile phone target extraction module is used to extract mobile phone target images from hand area images; Behavior model recognition unit: used for initial behavior recognition of surveillance video data; includes a multi-data fusion database, a dedicated behavior recognition model and a model training management module; wherein the multi-data fusion database is used to store image training data, the dedicated behavior recognition model is used to perform initial behavior recognition, and the model training management module is used to train and manage the dedicated behavior recognition model; Behavior logic recognition unit: used to perform secondary behavior recognition on surveillance video data; includes an eye image extraction module, an ear image extraction module and a behavior logic recognition module; wherein the eye image extraction module is used to extract the eye gaze direction from the personnel area image, the ear image extraction module is used to extract the ear image position from the personnel area image, and the behavior logic recognition module is used to perform secondary behavior recognition.

Images