A method, device, equipment and storage medium for call action recognition

Abstract

The present disclosure provides a call action recognition method, apparatus, device, and storage medium, which relate to the field of artificial intelligence technology, and particularly to technical fields such as image processing, deep learning, and computer vision. The specific implementation solution is as follows: perform region detection on the image to be detected to obtain the target image region where the person is located; match the target image region with the motion trajectory information to obtain a target motion trajectory that matches the target image region, where the information of the motion trajectory is: the information describing the motion trajectory of the same person determined according to the region of the person in the image; perform call action recognition on the target image region to obtain a recognition result; when the recognition result indicates that a call action is recognized, update the number of times the call action is recognized corresponding to the target motion trajectory; determine whether there is a real call action according to the updated number of times. Applying the above solution can improve the accuracy of call action recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technologies, and particularly to technical fields such as image processing, deep learning, and computer vision. Background Art

[0002] In working scenarios such as biochemical enterprises, if there is a situation where a person uses a mobile phone to make a call, it may cause a fire or affect the normal operation of electronics in the working scenario. Therefore, it is necessary to identify the call actions of people in the working scenario in order to timely detect the call behaviors of people in the working scenario and stop them. Summary of the Invention

[0003] The present disclosure provides a call action recognition method, apparatus, device, and storage medium.

[0004] On the one hand, the present disclosure provides a call action recognition method, including:

[0005] Performing region detection on the image to be detected to obtain a target image region where the person is located;

[0006] Matching the target image region with motion trajectory information to obtain a target motion trajectory that matches the target image region, where the motion trajectory information is: information describing the motion trajectory of the same person determined according to the region of the person in the image;

[0007] Performing call action recognition on the target image region to obtain a recognition result;

[0008] When the recognition result indicates that a call action is recognized, updating the number of times the call action is recognized corresponding to the target motion trajectory;

[0009] Determining whether there is a real call action according to the updated number of times.

[0010] On the other hand, the present disclosure provides a call action recognition apparatus, including:

[0011] An image region obtaining module, configured to perform region detection on the image to be detected to obtain a target image region where the person is located;

[0012] A motion trajectory obtaining module, configured to match the target image region with the motion trajectory information to obtain a target motion trajectory that matches the target image region, where the motion trajectory information is: information describing the motion trajectory of the same person determined according to the region of the person in the image;

[0013] A recognition result obtaining module, configured to perform call action recognition on the target image region to obtain a recognition result;

[0014] An action count update module, configured to update the count of recognized call actions corresponding to the target motion trajectory when the recognition result indicates that a call action is recognized;

[0015] A call action determination module, configured to determine whether a call action actually exists according to the updated count.

[0016] In another aspect of the present disclosure, there is provided an electronic device, including:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned call action recognition method.

[0020] In another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above-mentioned call action recognition method.

[0021] In another aspect of the present disclosure, there is provided a computer program product, including a computer program, and the computer program implements the above-mentioned call action recognition method when executed by a processor.

[0022] As can be seen from the above, in the solution provided by the embodiments of the present disclosure, after detecting the target image area where the person is located, the target motion trajectory of the person in the target image area is determined by matching the motion trajectory. In this way, when recognizing call actions in the target image area, counting can be performed based on the recognition result for different persons. Then, the above counting result is the count of the call actions detected for the same person in multiple frames of images. The larger this quantity is, the more times the person is detected with call actions, and the higher the probability that the person is actually on the phone. Therefore, the above counting result can more accurately reflect whether the person in the target image area is on the phone. Therefore, applying the solution provided by the embodiments of the present disclosure can not only successfully detect whether a person actually has a call action, but also improve the accuracy of detecting call actions.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0025] Figure 1 It is a schematic diagram of a scenario provided by an embodiment of the present disclosure;

[0026] Figure 2 It is a schematic flowchart of the first call action recognition method provided by an embodiment of the present disclosure;

[0027] Figure 3 It is a schematic flowchart of the first motion trajectory matching method provided by an embodiment of the present disclosure;

[0028] Figure 4 It is a schematic flowchart of the second call action recognition method provided by an embodiment of the present disclosure;

[0029] Figure 5 It is a schematic flowchart of the second motion trajectory matching method provided by an embodiment of the present disclosure;

[0030] Figure 6 It is a schematic structural diagram of a call action recognition device provided by an embodiment of the present disclosure;

[0031] Figure 7 It is a block diagram of an electronic device for implementing the call action recognition method provided by an embodiment of the present disclosure. Detailed implementation manners

[0032] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist in understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0033] An application scenario of an embodiment of the present disclosure is described below.

[0034] See Figure 1 , and the shooting location in the figure is the workplace. In the solution provided by the present disclosure, image acquisition devices, such as cameras, can be pre-set around the workplace to capture images of the workplace. The image acquisition devices can continuously capture images for a period of time to obtain images at multiple moments, Figure 1 is the image at the moment of 16:16 captured.

[0035] The solution provided by the embodiment of the present disclosure can be used to perform region detection on all the captured images to obtain the image region where the person is located in the image. See the image region where the square frame is located in Figure 1 . By performing call recognition on this image region, it can be determined whether the person in this image region is on a call.

[0036] The execution subject of the embodiments of the present disclosure may be an electronic device communicatively connected to the above-mentioned image acquisition device. The electronic device can receive the images captured by the image acquisition device and perform call action recognition based on the received images. For example, the above-mentioned electronic device may be a server, a desktop computer, etc.

[0037] The call action recognition method provided by the embodiments of the present disclosure will be described in detail below through specific embodiments.

[0038] In one embodiment of the present disclosure, refer to Figure 2 , a flowchart of the first call action recognition method is provided, and the method includes the following steps S201-S205.

[0039] Step S201: Perform region detection on the image to be detected to obtain the target image region where the person is located.

[0040] The target image region is determined by the position of the person in the image to be detected. The target image region may include a partial region of the person, for example, the head and shoulder region of the person, or may include the entire region of the person.

[0041] Two implementation manners for region detection to obtain the target image region will be described below.

[0042] In one implementation manner, the image to be detected may be input into a preset detection model to obtain the target image region containing the person. The detection model may be a target detection model pre-trained with sample images labeled with the region where the person is located. Specifically, the target detection model may be a target detection model based on yolov5, a target detection model based on yolo3D, etc.

[0043] In another implementation manner, the image features of the image to be detected may also be extracted, and then the person in the image to be recognized may be detected according to the extracted image features, so as to determine the target image region. For example, after extracting the image features, the connected regions in the image may be determined according to the image features, and then the region containing the above-mentioned connected regions may be used as the target image region containing the person.

[0044] Step S202: Match the target image region with the motion trajectory information to obtain the target motion trajectory that matches the target image region.

[0045] The motion trajectory of the person represents the motion state of the person, and can be determined according to the position sequence formed by the positions of the person in multiple frames of images.

[0046] Among them, the motion trajectory information is: information describing the motion state of the person determined according to the regions of the same person in different images. Each motion trajectory corresponds to the motion state of the same person, and different motion trajectories correspond to different motion states of different people.

[0047] Because the image acquisition device continuously captures images, the electronic device identifies the area where the person is located in each image captured by the image acquisition device. As images are continuously captured, different motion trajectories can be formed based on the identified area where the person is located. Each motion trajectory is continuously updated as images are continuously captured. The above motion trajectory is: the motion trajectory formed based on the previous image captured before the current image to be detected, that is, the motion trajectory obtained when performing area detection on the image to be detected. In other words, the motion trajectory mentioned in each step of the present embodiment is the obtained motion trajectory.

[0048] The target motion trajectory refers to the motion trajectory that matches the target image region. The matching relationship between the motion trajectory and the target image region can be determined based on the positional relationship between the target image region and the motion trajectory. For example, if the target image region is located on motion trajectory A or is close to motion trajectory A, then the target image region is considered to match motion trajectory A, and motion trajectory A is the target motion trajectory that matches the target image region.

[0049] Among them, when judging the positional relationship between the target image area and the motion trajectory, the motion trajectory can be used to predict the area where the person appears in the image to be detected, and then compared with the area position of the target image area to determine whether the target image area matches the motion trajectory. The specific implementation method can be found in the following Figure 3 The embodiment shown will not be described in detail here.

[0050] In another case, when judging the positional relationship between the target image area and the motion trajectory, a distance threshold can be preset, and a pixel point can be selected in the target image area. If the distance between the pixel point and the above-mentioned motion trajectory is less than the preset distance threshold, it is determined that the target image area is close to the motion trajectory, and thus it is determined that the target image area matches the motion trajectory.

[0051] Step S203: performing call action recognition on the target image area to obtain a recognition result.

[0052] In one embodiment of the present disclosure, a target image region can be input into a preset classification model to obtain the action type of the person in the target image region. Call action recognition can be achieved by determining whether the obtained action type belongs to a call action type. Specifically, the classification model can be a network model pre-trained using sample images labeled with action types. The network model can be a Resnet50 model, a MobileNet model, or the like. In this case, the accuracy of the action type results can be improved through iterative training of the network model.

[0053] In another embodiment of the present disclosure, the human key point positions of a person can be obtained from the target image region, including the hand position, the mouth position, etc. According to the relationship between the key point positions, it can be determined whether the person makes a call gesture, so as to judge whether the person is on a call.

[0054] The recognition result obtained after call recognition includes: representation information indicating that a call action is recognized or not recognized. In addition, the recognition result may also include: the probability indicating the accuracy of the above representation information, the rectangular region position indicating the position of the target object region, etc.

[0055] Step S204: When the recognition result represents that a call action is recognized, update the number of times the call action is recognized corresponding to the target motion trajectory.

[0056] In one embodiment of the present disclosure, updating the number of times the call action is recognized means increasing the value of the above number. For example, the initial value of the number of call actions recognized corresponding to the target motion trajectory can be set to 0. For the recognition results obtained by performing call action recognition on multiple frames of images to be detected according to the foregoing steps, for each frame of the image to be detected where a call action is recognized, the value of the above number corresponding to the target motion trajectory matching the target image region where the call action is recognized is increased by 1.

[0057] In another embodiment of the present disclosure, when updating the above number, the image quality of the image to be detected can also be considered, and the specific implementation method is as follows:

[0058] When the recognition result represents that a call action is recognized, perform image quality detection on the image to be detected to obtain a quality representation value for describing the quality of the image to be detected; if the quality representation value is greater than a preset representation threshold, update the number of times the call action is recognized corresponding to the target motion trajectory.

[0059] The recognition result represents that a call action is recognized, that is, there is representation information in the recognition result indicating that a call action is recognized.

[0060] The above quality representation value can be represented by the clarity of the image to be detected, and the clarity of the image to be detected can be determined based on whether there are situations such as blurring and backlighting in the image to be detected. If the obtained clarity is higher, the corresponding quality representation value is larger, and vice versa.

[0061] Alternatively, the image to be detected can also be input into a preset quality classification model to obtain a quality characterization value output by the quality classification model. The above quality classification model can be a network model pre-trained with sample images labeled with quality types. The network model can be a Resnet50 model, a MobileNet model, etc. When labeling the sample images, the sample images can be labeled as high-quality types and low-quality types, where the high-quality types are represented by larger quality characterization values and the low-quality types are represented by smaller quality characterization values. In this way, when the trained quality classification model classifies the quality of the image to be detected, it can output a quality characterization value as the quality classification result.

[0062] The above characterization threshold is a preset value. If the quality characterization value is greater than the characterization threshold, it indicates that the image quality of the image to be detected is relatively high. In this case, the number of recognized call actions corresponding to the target motion trajectory is updated.

[0063] As can be seen from the above, in the solution provided in this embodiment, when updating the number of recognized call actions, the counting is based on the recognition results of the images to be detected with relatively high image quality, that is, the quality characterization value is greater than the characterization threshold. The images to be detected with relatively high image quality are more accurate for the image information used for action recognition. Correspondingly, the accuracy of the recognition result and the counting result can be improved, and the number of false alarms caused by misdetecting call actions in the images to be detected can be reduced.

[0064] Moreover, by screening the images to be detected according to the image quality, the solution provided in this embodiment of the present disclosure can be applied to scenarios where low-quality images are easily obtained, such as scenarios with drastic changes in light, scenarios where the mobile phone is blocked or unclear, etc., and has good versatility.

[0065] Step S205: Determine whether there is a real call action according to the updated number.

[0066] The more the number of recognized call actions, that is, the more images to be detected with the recognition result of the existence of call actions, it can be understood that the image information of more images to be detected determines the existence of call actions, the reference image information is more sufficient, and the judgment of the existence of call actions is more accurate.

[0067] Specifically, it can be determined whether the number of recognized call actions in the first number of consecutive frames of images collected recently reaches a preset number according to the updated number; if so, it is determined that there is a real call action; otherwise, it is determined that there is no real call action.

[0068] The first quantity of frame images can be multiple frames of to-be-detected images that are continuously captured by an image acquisition device within a recent short time period, such as within 1 minute or within 30 seconds. For each to-be-detected image, the number of recognized call actions corresponding to the target motion trajectory can be updated according to the aforementioned steps S201 - S204. In this case, the number of recognized call actions accumulated for each motion trajectory can be obtained for comparison with a preset number. If the preset number is reached, it is determined that a call action truly exists; otherwise, no call action exists.

[0069] The preset number can be determined according to the number of the first quantity of frame images. For example, the preset number can be 80% of this quantity. That is, when there are 5 frames in the first quantity of frame images, if at least 4 frames recognize a call action, it is considered that the person in the to-be-detected image has a call action.

[0070] In this case, a relatively high proportion of the to-be-detected images in the first quantity of frame images are determined to have a call action. Since the first quantity of frame images are continuously captured images and the time interval for image acquisition is short, the actions made by the person within a short time are relatively stable. Thus, when performing call action recognition, if a sufficient number of recognition results all indicate that the person has made a call recently, it can be considered that the probability of the person having a call action is relatively high, further improving the accuracy of call action recognition.

[0071] As can be seen from the above, in the solution provided by the embodiments of the present disclosure, after detecting the target image area where the person is located, by matching the motion trajectory, the target motion trajectory of the person in the target image area is determined. In this way, when performing call action recognition on the target image area, counting can be performed based on the recognition results according to different persons. Then, the above counting result is the count of the call actions detected for the same person in multiple frames of images. The larger this quantity is, the more times the person is detected to have a call action, and the higher the probability that the person is truly on the phone. Therefore, the above counting result can more accurately reflect whether the person in the target image area is on the phone. So, applying the solution provided by the embodiments of the present disclosure can not only successfully detect whether a person truly has a call action, but also improve the accuracy of detecting call actions.

[0072] Next, the specific implementation manner of matching the target image area with the motion trajectory in the aforementioned step S202 will be described.

[0073] In an embodiment of the present disclosure, referring to Figure 3 , for the aforementioned step S202, the target image area can be matched with the information of each motion trajectory according to the following steps S301 - S304.

[0074] Step S301: Predict the motion state parameters of the person corresponding to the motion trajectory according to the state prediction parameters corresponding to the motion trajectory.

[0075] The motion trajectory can be obtained from the prior images captured prior to the current image to be detected. For example, a preset image acquisition device A can be used to continuously capture a work site to obtain a set of images to be detected with sequential shooting times. Among them, the current image to be detected is the Nth image arranged in the order of shooting times in the set of images to be detected, and the first N - 1 images are the above-mentioned prior images.

[0076] The state prediction parameters can be used to predict the motion state at a later time according to the motion state at a prior time on the motion trajectory. Specifically, the state prediction parameters can be obtained according to the Kalman filter coefficients, configure the corresponding time update function and measurement update function in the Kalman filter, and use the parameters of the motion state at the prior time as the input value to input the Kalman filter to obtain the predicted parameters of the motion state. Among them, the time update function is used to obtain the predicted value representing the motion state at a later time according to the parameters of the motion state at the prior time, and the measurement update function is used to correct the obtained predicted value to obtain the above-mentioned parameters of the motion state.

[0077] The above-mentioned parameters of the motion state are attribute parameters representing the motion state of the person, which may include the speed, acceleration, etc. of the person's motion.

[0078] Step S302: Determine the candidate image region where the person corresponding to the motion trajectory appears in the image to be detected according to the motion state parameters.

[0079] After obtaining the motion state parameters, according to the position of the person at the previous moment corresponding to the moment of the image to be detected in the motion trajectory and the motion process represented by the motion state parameters, the position where the person moves to in the image to be detected can be obtained; in this case, the image region including this position in the image to be detected is used as the candidate image region, that is, the region where the person may appear in the image to be detected.

[0080] Step S303: Extract the regional features of the target image region.

[0081] In an embodiment of the present disclosure, the feature vector corresponding to and describing the characteristics of the above image can be obtained according to the color, texture, and shape of the image in the target image region as the regional feature.

[0082] Step S304: If the regional features match the person features of the person corresponding to the motion trajectory and the position of the target image region matches the position of the determined candidate image region, determine that the motion trajectory is the target motion trajectory matching the target image region.

[0083] The personal characteristics of the person corresponding to the motion trajectory can be determined according to the first personal characteristics extracted from the area where the person is located in the image used to form the motion trajectory.

[0084] The above-mentioned personal characteristics can be obtained from the classification network model used in the reID (Re-identification) technology. This network model uses the ID (Identification) of the person as the annotation. The trained network model extracts the personal characteristics of the corresponding person through the image. There is an association between the obtained personal characteristics and the ID of the person corresponding to the motion trajectory. In this case, the ID of the person corresponding to the motion trajectory can be judged by the similarity between the above-mentioned first personal characteristics and the personal characteristics extracted by the classification network model, and the personal characteristics corresponding to this ID can be obtained accordingly.

[0085] In this case, the matching of the motion trajectory can be performed when there are disappearing people in some of the images to be detected. For example, some people may be walking in the images to be detected and thus go out of the shooting range, etc. In this case, the personal characteristics can still be confirmed through reID in the above process and used for matching with the regional characteristics.

[0086] In an embodiment of the present disclosure, it is possible to determine whether the regional characteristics match the personal characteristics according to the relationship between the first feature vector representing the regional characteristics and the second feature vector representing the personal characteristics. Specifically, the relationship between the above-mentioned first feature vector and the second feature vector can be represented by the distance or cosine similarity between the first feature vector and the second feature vector. The embodiments of the present disclosure do not limit this.

[0087] As can be seen from the above, in the solution provided by the embodiments of the present disclosure, when matching the target image area and the motion trajectory, not only the regional characteristics are matched with the personal characteristics of the person corresponding to the motion trajectory, but also the matching relationship between the positions of the target image area and the candidate image area is referred to. Multiple pieces of information are used to confirm whether the target image area matches the motion trajectory, and the reference information is relatively sufficient, so the obtained matching result is more accurate.

[0088] In one implementation manner, in the above step S304, if it is determined that the target motion trajectory fails, that is, the position of the target image area does not match the position of the candidate image area determined by any motion trajectory, or the regional characteristics do not match the personal characteristics of the person corresponding to any motion trajectory, then the intersection-over-union ratio between the target image area and the candidate image areas corresponding to each motion trajectory can be used to determine the target motion trajectory from the motion trajectories.

[0089] The above intersection over union (IoU) is used to represent the degree of overlap between the target image region and the candidate image region. The higher the degree of overlap, the greater the likelihood that the target image region matches the candidate image region, that is, the greater the likelihood of matching the motion trajectory of the predicted candidate image region. Specifically, a proportional threshold can be preset. If the IoU between the target image region and the candidate image corresponding to the motion trajectory X reaches the proportional threshold, it is considered that the target image region matches the motion trajectory X. In this case, even if the determination of the trajectory fails, the IoU between the target image region and the candidate image region can still be used to accurately match the target region and the motion trajectory.

[0090] In the above step S304, after obtaining the target motion trajectory, the state prediction parameters corresponding to the target motion trajectory can also be updated according to the target image region.

[0091] Different state prediction parameters correspond to different update methods. For example, when the state prediction parameter is a parameter obtained according to the Kalman filter coefficient, the predicted value and the measured value can be fused to obtain a fused value, and the updated Kalman filter coefficient can be obtained according to the fused value, thereby obtaining the updated state prediction parameter.

[0092] The predicted value is: a parameter representing the predicted position of the candidate image region; the measured value is: a parameter representing the position of the target image region

[0093] As can be seen from the above, updating the state prediction parameters enables the updated state prediction parameters to conform to the actual motion state of the person corresponding to the target image region in the image to be detected, improving the accuracy of the state prediction parameters. When the state prediction parameters are used for prediction again, the accuracy of the prediction results is also correspondingly improved.

[0094] In an embodiment of the present disclosure, matching the information of the target image region and the motion trajectory to obtain a target motion trajectory that matches the target image region includes:

[0095] Determining, from the motion trajectories, available motion trajectories that have been successfully matched with the image region continuously for the preset number of times; matching the information of the target image region and the available motion trajectories to obtain a target motion trajectory that matches the target image region.

[0096] In this embodiment, when using the motion trajectory for matching, in the continuously recently obtained images to be detected, based on the motion trajectories that have successfully matched the image regions for a preset number of times, the available motion trajectories can be obtained. Over a long period of time, the person may have completely left the area captured by the image acquisition device. For example, the staff may leave the workplace with the image acquisition device after work. At this time, no effective motion trajectory can be obtained from the captured images. Therefore, the images to be detected continuously captured in the recent time period are required as the source for obtaining the motion trajectories. Conversely, if the motion trajectory has successfully matched the image region for a preset number of times continuously recently, it indicates that the person corresponding to the motion trajectory has been in the captured area all the time. Sufficient image information can be obtained when performing the call action recognition on it, and the recognition result is relatively accurate.

[0097] The specific method of matching is the same as the steps in the foregoing Figure 3 embodiment, except that only the name concepts such as available motion trajectory and motion trajectory are replaced. Details are not described here again. In this case, the successful matching of the motion trajectory with a considerable number of image regions indicates that the obtained motion trajectory conforms to the motion state of the person described by a considerable number of image information, improving the accuracy of the obtained motion trajectory.

[0098] Next, the overall process of the call recognition method provided by the embodiments of the present disclosure will be described through Figure 4 and Figure 5 .

[0099] Figure 4 In , the head and shoulder area of the human body is the target image area obtained by detecting the image to be detected; Resnet head and shoulder classification indicates that the Resnet model performs call action recognition on the target image area to obtain the recognition result; the Resnet quality model is used to determine whether the image to be detected is a high-quality image; the recognition result accumulation represents the update of the number of times indicating the recognition of the call action in the recognition results of the high-quality images to be detected. Voting is used to determine whether the person is making a phone call, that is, to determine whether the person actually has a call action according to the updated number of times.

[0100] Deep Sort human head and shoulder tracking is used to match the target image area and the obtained motion trajectory to confirm the specific person to whom the recognition result belongs. Figure 5 The embodiment is a specific matching method.

[0101] See Figure 5 , Kalman filter prediction is used to obtain the motion state parameters of the person corresponding to the motion trajectory and predict the candidate image area. The specific implementation method refers to the foregoing steps S301 - S302, and details are not described here again.

[0102] Confirm that the motion trajectory corresponding to the Kalman filter is an available motion trajectory; if not confirmed, it means it is not an available motion trajectory, and then this motion trajectory is not used for prediction.

[0103] The available motion trajectory is used for cascaded matching with the target image region obtained by head and shoulder detection. For the specific implementation method, refer to the steps of the foregoing Figure 3 embodiment, which will not be elaborated here.

[0104] The successfully matched tracking indicates that there is a target motion trajectory that matches the target image region. In this case, Kalman filter update is performed, that is, the state prediction parameters corresponding to the target motion trajectory are updated according to the target image region.

[0105] The unmatched tracking and the unmatched detection indicate that the determination of the target motion trajectory fails. The unmatched tracking means that the target image region does not match any motion trajectory. The unmatched detection means that there is a motion trajectory that does not match any target image region. For example, in the image A to be detected, target image regions corresponding to 3 persons are detected, but the obtained motion trajectories are 4, then at least one motion trajectory does not match any target image region. In this case, and in the case where the motion trajectory is not an available motion trajectory, the intersection over union method in the foregoing embodiment is used for matching to obtain the target motion trajectory, that is, IOU Match (Intersection over Union Match) in the figure.

[0106] Corresponding to the foregoing call action recognition method, an embodiment of the present disclosure further provides a call action recognition device.

[0107] In an embodiment of the present disclosure, refer to Figure 6 for a structural schematic diagram of a call action recognition device, the device includes:

[0108] An image region obtaining module 601, configured to perform region detection on the image to be detected to obtain a target image region where a person is located;

[0109] A motion trajectory obtaining module 602, configured to match the target image region with the information of the motion trajectory to obtain a target motion trajectory that matches the target image region, where the information of the motion trajectory is: information describing the motion trajectory of the person determined according to the region of the same person in the image;

[0110] An identification result obtaining module 603, configured to perform call action recognition on the target image region to obtain an identification result;

[0111] An action times updating module 604, configured to update the number of times the call action is recognized corresponding to the target motion trajectory when the identification result indicates that a call action is recognized;

[0112] A call action determination module 605, configured to determine whether a call action actually exists according to the updated number of times.

[0113] As can be seen from the above, in the solution provided by the embodiments of the present disclosure, after detecting the target image area where the person is located, the target motion trajectory of the person in the target image area is determined by matching the motion trajectory. In this way, when performing call action recognition on the target image area, counting can be performed based on the recognition result for different persons. Then, the above counting result is the count of the call actions detected for the same person in multiple frames of images. The larger this quantity is, the more times the person is detected with a call action, indicating that the probability of the person actually making a call is higher. Therefore, the above counting result can more accurately reflect whether the person in the target image area is making a call. Therefore, applying the solution provided by the embodiments of the present disclosure can not only successfully detect whether a person actually has a call action, but also improve the accuracy of detecting call actions.

[0114] In an embodiment of the present disclosure, the action number update module 604 is specifically configured to, when the recognition result indicates that a call action is recognized, perform image quality detection on the image to be detected to obtain a quality characterization value for describing the quality of the image to be detected; if the quality characterization value is greater than a preset characterization threshold, update the number of times the call action is recognized corresponding to the target motion trajectory.

[0115] As can be seen from the above, in the solution provided by this embodiment, when updating the number of times a call action is recognized, counting is performed based on the recognition result of the image to be detected with a quality characterization value greater than the characterization threshold, that is, an image with relatively high image quality. The image information used for recognition in the image to be detected with relatively high image quality is more accurate. Correspondingly, the accuracy of the recognition result and the counting result can be improved, and the number of false alarms caused by misdetecting call actions in the image to be detected can be reduced.

[0116] Moreover, by screening the images to be detected based on the image quality, the solution provided by the embodiments of the present disclosure can be applied to scenarios where it is easy to obtain low-quality images, such as scenarios with drastic changes in light, scenarios where the mobile phone is blocked or not clearly visible, etc., and has good versatility.

[0117] In one embodiment of the present disclosure, the motion trajectory obtaining module 602 is specifically configured to match the target image region with the information of each motion trajectory in the following manner: predicting the motion state parameters of the person corresponding to the motion trajectory according to the state prediction parameters corresponding to the motion trajectory; determining the candidate image region where the person corresponding to the motion trajectory appears in the image to be detected according to the motion state parameters; extracting the region features of the target image region; and if the region features match the person features of the person corresponding to the motion trajectory and the position of the target image region matches the position of the determined candidate image region, determining the motion trajectory as the target motion trajectory matching the target image region.

[0118] As can be seen from the above, in the solution provided by the embodiment of the present disclosure, when matching the target image region with the motion trajectory, not only the region features are matched with the person features of the person corresponding to the motion trajectory, but also the matching relationship between the positions of the target image region and the candidate image region is referred to. Multiple pieces of information are used to confirm whether the target image region matches the motion trajectory, and the reference information is relatively sufficient, so the obtained matching result is more accurate.

[0119] In one embodiment of the present disclosure, in the case where the determination of the target motion trajectory fails, the device further includes:

[0120] A motion trajectory determination module, configured to determine the target motion trajectory from the motion trajectories according to the intersection-over-union ratio between the target image region and the candidate image regions corresponding to the motion trajectories.

[0121] In this case, even when the determination of the trajectory fails, the target region and the motion trajectory can still be accurately matched by the degree of overlap between the target image region and the candidate image regions.

[0122] In one embodiment of the present disclosure, after obtaining the target motion trajectory, the device further includes:

[0123] A parameter update module, configured to update the state prediction parameters corresponding to the target motion trajectory according to the target image region.

[0124] As can be seen from the above, updating the state prediction parameters enables the updated state prediction parameters to conform to the actual motion state of the person corresponding to the target image region in the image to be detected, improving the accuracy of the state prediction parameters. When using the state prediction parameters for prediction again, the accuracy of the prediction result is also correspondingly improved.

[0125] In one embodiment of the present disclosure, the motion trajectory obtaining module 602 is specifically configured to determine, from the motion trajectory, available motion trajectories that have been successfully matched with the image region for a preset number of consecutive times; and match the information of the target image region with the available motion trajectories to obtain a target motion trajectory that matches the target image region.

[0126] In this case, the motion trajectory is successfully matched with a considerable number of image regions, indicating that the obtained motion trajectory conforms to the motion state of the person described by a considerable amount of image information, thereby improving the accuracy of the obtained motion trajectory.

[0127] In one embodiment of the present disclosure, the call action determining module 605 is specifically configured to determine, according to the updated number of times, whether the number of times the call action is recognized in the first number of consecutive frames of images collected recently has reached a preset number; if so, it is determined that there is a real call action; otherwise, it is determined that there is no real call action.

[0128] In this case, a relatively high proportion of the images to be detected in the first number of frames of images are determined to have a call action, and the first number of frames of images are all continuously collected images with a short time interval between image collections. The actions made by the person in a short time are relatively stable. When performing recognition, if a sufficient number of recognition results all indicate that the person is on the phone recently, based on the consistency of the recognition results, it can be considered that the probability of the person having a call action is relatively high, further improving the accuracy of call action recognition.

[0129] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the person image information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0130] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0131] In one embodiment of the present disclosure, an electronic device is provided, including:

[0132] At least one processor; and

[0133] A memory communicatively connected to the at least one processor; wherein,

[0134] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the call action recognition method described in any of the foregoing embodiments.

[0135] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the call action recognition method described in any of the foregoing embodiments.

[0136] In one embodiment of the present disclosure, a computer program product is provided, including a computer program which, when executed by a processor, implements the call action recognition method described in any of the foregoing embodiments.

[0137] Figure 7 A schematic block diagram of an exemplary electronic device 700 that can be used to implement the embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0138] As Figure 7 shown, the device 700 includes a computing unit 701 which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0139] A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0140] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as the call action recognition method. For example, in some embodiments, the call action recognition method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the call action recognition method described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to execute the call action recognition method by any other suitable means (e.g., by means of firmware).

[0141] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special or general-purpose programmable processor, that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0145] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, can also be a server of a distributed system, or a server incorporating a blockchain.

[0147] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitations are imposed herein.

[0148] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A call action recognition method, comprising: Performing region detection on the image to be detected to obtain a target image region where the person is located; Matching the target image region with motion trajectory information to obtain a target motion trajectory that matches the target image region, wherein the motion trajectory information is: information describing the motion trajectory of the same person determined according to the region of the person in the image, the motion trajectory of the person is determined according to the position sequence formed by the positions of the person in multiple frames of images, the matching relationship between the motion trajectory and the target image region is determined by the position relationship between the target image region and the motion trajectory, each motion trajectory corresponds to the motion state of the same person, and different motion trajectories correspond to the motion states of different persons; Performing call action recognition on the target image region to obtain a recognition result; When the recognition result indicates that a call action is recognized, performing image quality detection on the image to be detected to obtain a quality characterization value for describing the quality of the image to be detected; If the quality characterization value is greater than a preset characterization threshold, updating the number of times the call action is recognized corresponding to the target motion trajectory; Determining whether there is a real call action according to the updated number of times.

2. The method according to claim 1, wherein The step of matching the target image region with the motion trajectory information to obtain a target motion trajectory that matches the target image region includes: Matching the target image region with each piece of motion trajectory information in the following manner: Predicting the motion state parameters of the person corresponding to the motion trajectory according to the state prediction parameters corresponding to the motion trajectory; Determining a candidate image region where the person corresponding to the motion trajectory appears in the image to be detected according to the motion state parameters; Extracting the region features of the target image region; If the region features match the person features of the person corresponding to the motion trajectory and the position of the target image region matches the position of the determined candidate image region, determining the motion trajectory as the target motion trajectory that matches the target image region.

3. The method according to claim 2, further comprising, when the determination of the target motion trajectory fails: Determining the target motion trajectory from the motion trajectories according to the intersection-over-union ratio between the target image region and the candidate image regions corresponding to each motion trajectory.

4. The method according to claim 2, further comprising, after obtaining the target motion trajectory: Updating the state prediction parameters corresponding to the target motion trajectory according to the target image region.

5. The method according to claim 1, wherein The step of matching the target image region with the information of the motion trajectory to obtain a target motion trajectory that matches the target image region includes: Determining available motion trajectories that have been successfully matched with the image region continuously for the most recent preset number of times from the motion trajectories; Matching the target image region with the information of the available motion trajectories to obtain a target motion trajectory that matches the target image region.

6. The method according to claim 1, wherein The step of determining whether there is a real call action according to the updated number of times includes: Judging whether the number of times the call action is recognized in the first number of frames of images collected continuously most recently has reached a preset number according to the updated number of times; If it is yes, it is determined that there is a real call action; Otherwise, it is determined that there is no real call action.

7. A call action recognition device, comprising: An image area obtaining module, configured to perform area detection on a to-be-detected image to obtain a target image area where a person is located; A motion trajectory obtaining module, configured to match the target image area with information of a motion trajectory to obtain a target motion trajectory that matches the target image area, where the information of the motion trajectory is: information describing the motion trajectory of the same person determined according to the area of the person in the image, the motion trajectory of the person is determined according to a position sequence formed by the positions of the person in multiple frames of images, the matching relationship between the motion trajectory and the target image area is determined by the position relationship between the target image area and the motion trajectory, each motion trajectory corresponds to the motion state of the same person, and different motion trajectories correspond to the motion states of different persons; A recognition result obtaining module, configured to perform call action recognition on the target image area to obtain a recognition result; An action count updating module, specifically configured to, when the recognition result indicates that a call action is recognized, perform image quality detection on the to-be-detected image to obtain a quality characterization value for describing the quality of the to-be-detected image; if the quality characterization value is greater than a preset characterization threshold, update the count of the recognized call action corresponding to the target motion trajectory; A call action determination module, configured to determine whether there is a real call action according to the updated count.

8. The device according to claim 7, wherein The motion trajectory obtaining module is specifically configured to match the target image area with the information of each motion trajectory in the following manner: predict the motion state parameters of the person corresponding to the motion trajectory according to the state prediction parameters corresponding to the motion trajectory; determine the candidate image area where the person corresponding to the motion trajectory appears in the to-be-detected image according to the motion state parameters; extract the area features of the target image area; if the area features match the person features of the person corresponding to the motion trajectory and the position of the target image area matches the position of the determined candidate image area, determine the motion trajectory as the target motion trajectory that matches the target image area.

9. The device according to claim 8, in the case of failure to determine the target motion trajectory, the device further comprises: A motion trajectory determination module, configured to determine the target motion trajectory from the motion trajectories according to the intersection-over-union ratio between the target image area and the candidate image areas corresponding to each motion trajectory.

10. The device according to claim 8, after obtaining the target motion trajectory, the device further comprises: A parameter updating module, configured to update the state prediction parameters corresponding to the target motion trajectory according to the target image area.

11. The device according to claim 7, wherein The motion trajectory acquisition module is specifically configured to determine, from the motion trajectories, available motion trajectories that have been successfully matched with the image region for a preset number of consecutive times; match the information of the target image region with the available motion trajectories to obtain a target motion trajectory that matches the target image region.

12. The apparatus according to claim 7, wherein, The call action determination module is specifically configured to determine, according to the updated number of times, whether the number of times the call action is recognized in the first number of consecutive frames of images collected recently has reached a preset number of times; If so, it is determined that there is a real call action; Otherwise, it is determined that there is no real call action.

13. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, where the computer program, when executed by a processor, implements the method according to any one of claims 1-6.

Images